An Object Based Auto Annotation Image Retrieval System

Transcription

1 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) An Obect Based Auto Annotaton Image Reteval System Pe-Cheng Cheng, Been-Chan Chen 2, Hao-Ren Ke, and We-Pang Yang, 4 Depatment of Compute & Infomaton Scence, atonal Chao Tung Unvesty, 00 Ta Hsueh Rd., Hsnchu, Tawan 0050, R.O.C. 2 Depatment of Compute Scence and Infomaton Engneeng, atonal Unvesty of Tanan,, Sec. 2, Su Lne St., Tanan, Tawan 70005, R.O.C. Lbay and Insttute of Infomaton Management, atonal Chao Tung Unvesty, 00 Ta Hsueh Rd., Hsnchu, Tawan 0050, R.O.C. 4 Depatment of Infomaton Management, atonal Dong Hwa Unvesty, Sec. 2, Da Hsueh Rd., Shou-Feng, Hualen, Tawan 9740, R.O.C. Abstact: In ths pape, we poposed an auto annotaton mage eteval system. In ou system, an mage was segmented nto egons, each of whch coesponds to an obect. The egons dentfed by egon-based segmentaton ae moe consstent wth human cognton than those dentfed by block-based segmentaton. Accodng to the obect s vsual featues (colo and shape), new obects wll be map to the smla clustes to obtan ts assocated semantc concept. The semantc concepts deved by the tanng mages may not be the same as the eal semantc concepts of the undelyng mages, because the fome concepts depend on the low-level vsual featues. To ameloate ths poblem, we popose a elevance-feedback model to lean the long-tem and shot-tem nteests of uses.the expements show that the poposed algothm outpefoms the tadtonal co-occuence model about 9.5%; futhemoe, afte fve tmes of elevance feedback, the mean aveage pecson mpoves fom 46% to 62.7%. Keywods: keywod-based mage eteval, co-occuence model, elevance feedback.. Intoducton The man methods of cuent mage eteval systems ae quey by example and quey by keywod. When usng the system of quey by example, t s necessay fo uses to offe an mage as an example fo quey [5] [8]. Some systems howeve offes smple tools of dawng that uses can daw out the fame of mage needed to quey []. Afte ths, the system wll extact the mage s featues and contast the smlaty of ts featues wth all othe mages n the mage database. Then, the mages wth hghly smlates n the database wll be output to uses. onetheless, ths method of quey s not convenent fo uses; t s had fo uses to daw an mage. On the othe hand, thee ae two man poblems wth tadtonal quey by keywod. One s t takes lots of tme and man foce when dong mage annotaton fo a lage amount of mage data. The second s that t s not easy to descbe the content of an mage. An mage obseved by dffeent people wll tgge dffeent feelngs, so when mage annotaton made by dffeent people, the annotaton wll be dffeent as well. It s a poblem wth subectvty. Ths pape poposes an auto annotate system to help explan the concept of mages and povdes semantc quey. We apply machne leanng and patten ecognton to help establsh mage annotaton, and poposed an obected based mage eteval system. Image was segment n to seveal homogeneous egons. The acqued egon by mage segmentaton s egaded as the obect exsted n the mage and these obects wll be accompaned by wods added atfcally to fom a tanng data set. Fo mages wthout mage annotaton, the system wll lean the semantc concept of nemages and ceate annotatons fo these nemages. By dong so, uses can quey mages by keywod whch s famla by uses. Afte auto annotated mechansm, we also allow use dong elevance feedback to ase the accuacy. In the secton 2, we wll evew the elated woks of mage eteval systems. Sectons descbe ou poposed obect based mage eteval system.

2 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) Secton 4 shows the expements. Secton 5 has a concluson of ou wok. 2. Related woks The content of mages s composed of obects exsted n mages, so t should be assued fst about what obects exst n mages when dong automatc mage annotaton. Fo the tme beng, obects n mages ae segmented by mage segmentaton and these obects fom mage segmentaton, one s block based [0] [] and the othe s egon based [2] [4] [7]. In pactce, block based s ease than egon based fo block based s only segment mages nto the same sze ectangles and fom obects by one o many ectangles. But obects segmented by block based ae dffeent fom what people can usually ecognze. Fo example, a pctue of a tge could be segmented nto seveal blocks by block based, but t s dffeent fom a tge n peoples mnds. Regon based can segment an mage nto an obect elatvely accepted by people, but egon based s moe dffcult. Though many mage segmentaton technques ae poposed, t s stll had to dentfy whch segmentaton method s the best, but the obects segmented by t ft bette wth people s mpesson of a peson. Afte acqung obects of mages, a system wll nfe possble semantc concepts and make mage annotaton though low level vsual featues of these obects (such as colo, shape, textue, and etc.). In the auto annotaton phases, Mo [] has poposed a co-occuence model and calculates the fequency of keywods and obects n the same cluste. In [4], t povdes a tanslaton model. It egads the tanslaton model as a dctonay and uses the machne tanslaton to tanslate the obects of mages nto anothe language. In [7], t ponts out that an obect has no equvalence wth a specfc wod. Some wods sometmes wll occu only when many obects co-occu. Fo nstance, when landscape s enteed as a keywod, the pctue of landscape wll appea only when obects of mountan, ves, falls, and felds co-occu. Fo elevant studes nowadays, thee ae thee mao epesentatons of helpng mages establsh annotaton: Fxed Annotaton Model [7], Semantc etwok Model [], and Pobablty Annotaton Vecto Model [7]. Fxed Annotaton Model epesent mage annotaton s to match a keywod wth an mage when the keywod and a cetan mage have hgh elevance. Semantc etwok Model adopts semantc netwok establshed by mages and keywods to epesent the semantc concept of mages. Pobablty Annotaton Vecto Model, pobablty vecto s used to epesent the semantc concept of mages. Evey element of vectos epesents a keywod ndcatng the possblty of the occuence of an mage.. Modfed Co-occuence Regon Based Model (MCORM) In ths pape, an appoach of automatc mage annotaton and quey s poposed whch s called as MCORM (Modfed Co-occuence Regon Based Model). Ths model mpoves and adusts the co-occuence model by the followng: () Acque obects by Regon Based nstead of Block Based n []. (2) An obect wll have a pobablty to map nto smla clustes to get moe specfed semantc concepts. () Emphasze moe on the mpotance obect of an mage. And n the Relevance Feedback pocess, t combnes uses concepts and habts to ase the accuacy of quey. Make use of ths system can avod the poblems caused by mage annotaton made atfcally. Thee ae thee models n ths system: Tanng Module, Annotaton Module and Quey and Feedback Module.. Tanng Module Ths pape use colo and shapes as mage vsual featues of mages. Befoe gettng the colo featues of two egons, ths system wll fnd a basc ectangle whch can enclose the whole egon and vews the mage egon as a sub-mage. Afte ths, the colo contast lst n [9] wll be adopted to quantfy the colo of all dots nto twenty fve colos. Afte quantfyng the colo, the colo featues acqung method n [] wll be used to get the featues n the basc ectangle and do the calculaton of colo smlaty. In tems of the shapes, the smlaty of shapes wll be calculated by the method mentoned n [6]. The fomula of calculatng smlaty of two obects n mages s showed as Eq.(): sm _ c, and sm _ s, ae the smlaty of colo and shape of two obects espectvely. Hee, wc and ws ae the weghts of colo and shape espectvely. sm, = wc sm _ c, + ws sm _ s, () Thee ae thee man steps of the tanng module n ths pape:. mage segmentaton; 2. K-Means Clusteng [2];. Infe the fequency of keywods n each cluste. The tanng pocess s as follows: fst, make annotaton fo all mages n the tanng

3 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) data base; second, Segment all mages n the tanng data base and get the egons of mages; thd, All egons segmented fom mage I fom the tanng database wll nhet all annotaton of I as ts own annotaton; fouth, Take out all the featues whch had been segmented wth the method mentoned; ffth, Cluste all egons wth K-Means Clusteng; sxth, Calculate the pobablty of evey keywod of all clustes. In step 6, the fomula used to calculate the pobablty of keywod w appeaed n Cluste K s as Eq.(2): w P K, M (2) w w s the tmes of keywod appeaed n a cetan cluste; M s the numbe of all keywods..2 Annotaton Module In tadtonal co-occuence module, the acqued obect s coesponded to the most smla cluste n the tanng module. In vew of that ths way s qute nappopate fo ts extemeness. Ths pape does nomalzatons of the latest clustes of the obects. The pobablty of keywod w appeaed n P, egon can be obtaned by Eq.(), whee s the coespondence smlaty wth cluste n egon ; P C, s the pobablty of keywod w appeaed n cluste of egon. P, = = s P s C, () Besdes, because the cente obects n an mage would be consdeed the theme o mpotant n ths mage, f egon s n the centod of mage I, the weght of ts semantc concept wll be hghlghted. At last, use Eq.(4) to calculate the pobablty of keywod w appeaed n mage I. s the numbe of egons n mage I; P, α s s the weght of mage I n egon. w s the pobable pobablty of keywod n egon. In pactce n ths pape, f egon s the centod egon n the mage, then α s set as. nstead of. When usng fomula (Eq.4), any keywod n whch egon that w has the bggest pobablty of occuence wll be ecoded and so as ts egon d. Ths ecod s called max_pw, whch s used to ecod the egon epesentng keywod n the whole mage and can be used fo use elevance feedback. P( I w ) = α P, w (4) = The way to epesent max_ps ( mage, wod, d ) whee mage s the name of ths mage. If the pobablty of keywod w appeang n all egons s 0, then d of ( mage, wod, d ) s -. Annotaton nfomaton fo an mage I as follows:. Quey and Feedback Module Afte fnshng automatc mage annotaton, thee s a coelaton weght of evey keywod n the dctonay and evey mage. When quey by keywod s used, uses can ente many keywods to quey. Due to t s had to acheve 00% accuacy by automatc makng annotaton wth machne leanng, ths pape poposed use elevance feedback to ase effcency and effects. On the othe hand, the nfomaton adopted n the tanng module could be dffeent fom the mage that needs annotaton and lmts the leanng effect, so we also defne and ceate a Cluste-Keywod Assocaton Map-CKAM. CKAM collects the feedback nfomaton of uses quey to adust the coelaton between clustes and keywods and ases the quey effcency of quey system. The coelatons between clustes and wods n CKAM ae leaned fom uses, so afte usng the system fo a whle, the coelaton nfomaton can eplace the KPV of each cluste n pevous tanng module. Hee, wcs, s the weght of cluste and keywod though feedback. At the begnnng, all elements on the lst ae set as. w w 2 L w n- w n c wcs, wcs,2 L wcs, n wcs, n c2 wcs2, wcs2,2 L wcs2, n wcs2, n M L L L L L ck wcsk, wcsk,2 L wcsk, n wcsk, n c k wcsk, wcsk,2 L wcsk, n wcsk, n Fgue : Cluste-Keywod Assocaton Map-CKAM When uses use quey by keywod, the system wll send nfomaton though CKAM to assess whch clustes of egons have hgh coelaton wth the keywod and the system wll povde mpotant nfomaton of calculatng the coelaton of keywods and mages fo queyng mage. When ' ' ' W = w w,..., w to uses ente a keywod set { }, 2 m

4 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) quey, the system wll calculate the coelaton of any mage I n mage database and these keywods W by fomula Eq.(5). m ' SIM ( I, W ) = β P( I ' ) + γ SIM (, w ) w m = (5) Thee ae two pats of Eq.(5). Fst s the coelaton of I and W calculated by automatc annotaton module. Second s the coelaton of egons of mages and keywods enteed by uses P I ' s the pobablty of though CKAM. Hee, ( ) ' the -th quey keywod appeas n mage I; ( ) SIM, s the coelaton of keywod and the numbe egon n mage I; β and γ ae weghts of these two pats espectvely. Though CKAM, whch knds of egons ae hghly coelated wth ths keywod can be known. Hee, Eq.(6) s the fomula of SIM (, ) n Eq.(5), and wcs, k s the weght of keywod and cluste k n CKAM; s x s the smlaty wth the x-th cluste n Regon ; wsc, c s the weght of keywod w y n n CKAM and the y-th cluste wth smlaty n Regon. s y wsc, c y y= SIM ( ) =, w (6) k wcs, k sx x= Though the calculaton of Eq.(5), the smlates of keywods enteed by uses and all mages can be acqued and the mage wth hgh smlaty can be sent back to uses though sotng. The mages though the quey pocess could be ncoect o not the one that uses want, theefoe use elevance feedback mechansm s poposed to ase the effcency of ths system. Thee ae thee pats of ths feedback module.. Modfy the pobablty of the occuence of keywods appeaed n assgned mages. 2. Modfy CKAM.. Send back new quey esults to uses. In the pocess of modfyng the pobablty of the occuence of keywods appeaed n assgned mages s to collect all mages of postve and negatve examples that uses send back and adust the pobablty of mages and keywods. Thee ae two pats n ths pocess:. Adust the pobablty of keywods of postve mages. 2. Adust the pobablty of keywods of negatve mages. Adust the pobablty of keywod n KPV accodng to the keywod w enteed by uses and the coespondng postve mage I. The way of adustng s to add c to the ognal pobablty. All values of keywods appeaed n mages s between 0 to, so afte addng, they system wll check f the value s ove, f so, then the value wll be. Smlaly, the system wll adust the pobablty of keywod n KPV accodng to the keywod w enteed by uses and the coespondng negatve mage I. The way of adustng s to subtact c 2 fom the ognal pobablty. Afte subtactng c 2, check f the value s smalle than 0; f so, then let the value be 0 and set the d of max_pw whee the keywod s coespondng to mage I -. Though modfyng CKAM, the type of egon of keywods of quey whch s moe accepted by uses can be acqued and can be used as efeence nfomaton fo the next quey. In ths pocess, fo the keywod n a keywod set, the system wll collect elated mages ecognzed by all uses and get the max_pw of all elevant mages whch ae elevant to the quey keywods and fnd out the egon whch can epesent the mage coespondng to the keywod w. The d of max_pw n some mages assgned by uses s set as -, whch means mages wthout any egons coespondng to the keywod s moe epesentable. In ths pocess, thee ae two steps. Fst, get all postve mages that the ds coespondng to the keywod n max_pw ae not equvalent to -,and add ts featues of egon to Local Domnate Regon Lst (LDRL) and add ts egon d to Global Domnate Regon Lst (GDRL), and adust CKAM. Second, to any postve mage I p, that ts d coespondng to the keywod w n max_pw equvalent to -, get the egon whch has the bggest smlaty wth the egon featue of LDRL n I p and set the d coespondng to the keywod w of max_pn I p nto the d of egon and adust CKAM. The pocess of adustng CKAM though a postve mage s as follow:. Get the egon that ts seal numbe s the d of max_pw coespondng to the keywod n mage I. 2. Get the latest cluste seal numbe ( c, c2, c ) and coespondng smlaty ( s, s2, s ) of the egon that ts seal numbe s d.. Add these thee smlaty values on coespondng wcs c, w n CKAM. Hee, wcs c, w s the value coespondng to the keywod n -th smla cluste seal numbe of egon n CKAM.

5 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) Afte ths, the system wll ecalculate the coelaton between all mages and feedback nfomaton. The pocess of any mage I though use elevance feedback and the smlaty of quey set W and I can be known by Eq.(7). Hee, u and v ae constants; m s the numbe of keywods n keywod ' P s the pobablty of the keywod n I w W; ( ' ) mage I. G s the egons set that n GDRL whle modfyng CKAM; G s the numbe of egons n G; s the smlaty between egon n S, G k G k mage I and the k-th egon n the whole G. SIM m ( I, W ) = u P( I ) + ' v max S w G = to, k m G k= (7) Though the pocess of use elevance feedback to modfy CKAM and afte usng ths system fo a whle, the coelaton between wods n CKAM and each cluste can be moe accepted by uses. So, the nfomaton can be used to eplace each KPV n the P, tanng database. The pobablty K appeaed n cluste K of any keywod can be modfed by Eq.(8). Hee, t s the tmes of keywod w has been queed; wcs K, s the value coespondng to the keywod w and the cluste CK n CKAM. wcsk, t P K, = (8) n wcsk, t 4. Expement In ths secton, the pactce of the system wll be ntoduced and the effcency of ths system wll also be evaluated. To solve the poblems that uses mght encounte whle dong the mage quey, the multple mage quey system s poposed n ou system. The nteface of ths system s shown n Fgue 2. Thee ae thee ways of quey n ths system: Quey by example, Quey by obect and Quey by keywod. G Fgue 2: Seach fall by quey by obect In ths expement, thee ae 200 mages n total. 800 mages of the 200 ae used fo the data of the tanng data set and the est 400 mages ae used fo testng. In the pocess of tanng, to the mages n the tanng data set, one to seven keywods ae gven to make mage annotaton. Afte makng annotaton, deleted those keywods not often appeang fst and then thee ae 64 keywods can be used fo makng automatc mage annotaton. When put MCORM nto pactce, K-Means s the tool of clusteng n ths pape and K s set as 00. Eq.() s used to calculate the smlaty of egons. In pactce, wc of Eq.() s set as 0.8, and ws s set as 0.4. In Eq.(), the ange of colo smlaty s between 0 and 2 and the shape smlaty s between 0 and, so the value of smlaty s between 0 and 2. In ths expement, 6 keywods ae used to test. Ths pape s to make mage annotaton by evsng the tadtonal co-occuence model and Regon Based mage segmentaton. So, n ode to compae the effcency of Regon Based and Block Based mage segmentaton whle usng co-occuence model, besdes MCORM, thee ae also two othe systems put nto pactce to evaluate the effcency was mplemented: () BCOM (Block Based Co-occuence Model): Ths model uses block based mage segmentaton to acque obects n mages and make annotaton fo mages by co-occuence model. (2) RCOM (Regon Based Co-occuence Model) : Ths model uses egon based mage segmentaton to acque obects n mages and make annotaton fo mages by co-occuence model. 2 to 5 keywods n each categoy ae taken fo testng. Table 2 s the whole MAP of above appoaches.

6 Poceedngs of the 5th WSEAS Intenatonal Confeence on Telecommuncatons and Infomatcs, Istanbul, Tukey, May 27-29, 2006 (pp509-54) BCOM RCOM MCORM MAP 25.55% 6.84% 46.00% Table : The whole MAP of above appoaches Fom above expement esults, t s obvous that n oveall effcency, MCORM s the best, RCOM s the second and BCOM s the wost. The obects acqued by Regon Based mage segmentaton ae bette than though Block Based n tems of the vsual acceptance to people. Theefoe, the effects made by RCOM ae bette than BCOM. MCORM gets the obects of mages by Regon Based and though above modfcaton poposed n ths pape, t ndeed ases the effcency of the system as we can see fom the expement esults. At last, quey s made though the keywods n Table 2 and evaluaton s also conducted to see whethe let system lean by nteactng wth uses can ase the effcency of the system o not. In the pocess of evey quey, uses can decde whethe the esult fom the system s elevant wth the obectve o not and uses wll send ths nfomaton back to the system. Use Relevance Feedback can ase effcency of the whole system and afte the system leans fom the elevant feedback fom uses fo 4 to 5 tmes, the advancement of the system s gettng smooth and eaches a good effcency level. Afte the ffth feedback, the oveall map of the system has ased fom 46.00% to 62.70%. 5. Conclusons and Futue Wok Ths pape poposed a Modfed Regon Based Co-occuence Model (MCORM) whch evsed fom []. In the expement n Chapte 4, t shows quey n the sx man categoes by RCOM s bette than BCOM by.29% n the MAP. MCORM has bette effcency than BCOM and RCOM. It pogesses by 20.45% and 9.6% espectvely n the MAP. Use Relevance Feedback poposed n ths pape has also made achevements n asng the effcency of the system. Ove all, afte usng the feedback system, the value has ased fom 46% to 62.7% n the MAP. Refeence [] A. D. Bmbo and P. Pala, Vsual Image Reteval by elastc matchng of use sketches, IEEE Tansactons on Patten Analyss and Machne Intellgence, Vol. 9, Issue 2, 997, pp [2] C. Cason, M. Thomas, S. Belonge, J. M. Hellesten, and J. Malk, Blobwold: A system fo egon-based mage ndexng and eteval, In Thd Intenatonal Confeence on Vsual Infomaton Systems, Lectue otes n Compute Scence, 999, pp [] L. Cnque Colo-based mage eteval usng spatal-chomatc hstogams, Image and Vson Computng, Vol. 9, Issue, 200, pp [4] P. Duygulu, K. Banad,. D. Fetas, and D. Fosyth, Obect ecognton as machne tanslaton: Leanng a lexcon fo a fxed mage vocabulay, In Seventh Euopean Confeence on Compute Vson, 2002, pp [5] M. Flckne, H. Sawhney, W. black, and J. Ashley, Quey by Image and Vdeo content: The QBIC System, IEEE Compute, Vol.28, Issue 9, 995, pp [6] D. Zhang and G. Lu, Impovng eteval pefomance of Zenke moment descpto on affned shapes, IEEE Intenatonal Confeence on Multmeda and Expo, vol., 2002, pp [7] J. Jeon, V. Lavenko, and R. Manmatha, Automatc Image Annotaton and Reteval usng Coss-Meda Relevance Models, ACM Confeence on Reseach and Development n Infomaton Reteval, 200, pp [8] J. R. Smth and S. F. Chang, Vsualseek: a fully automated content-based mage quey system, In Poceedngs of ACM Multmeda, 996, pp [9] K. C. Ravshanka, B. G. Pasad, S. K. Gupta, and K. K. Bswas, Domnant colo egon based ndexng fo cb, Intenatonal Confeence on Image Analyss and Pocessng, 998, pp [0] J. H. Lm, Leanable vsual keywods fo mage classfcaton, Poceedngs of the fouth ACM confeence on Dgtal lbaes, 999, pp [] Y. Lu, C. Hu, X. Zhu, H. J. Zhang, and Q. Yang, A Unfed Famewok fo Semantcs and Featue Based Relevance Feedback n Image Reteval Systems, ACM Multmeda, 2000, pp. -7. [2] J. McQueen, Some methods fo classfcaton and analyss of multvaate obsevatons, Poc. of the Ffth Bekeley Symposum on Mathematcal Statstcs and Pobablty, 967, pp []Y. Mo, H. Takahash, and R. Oka, Image-to-wod tansfomaton based on dvdng and vecto quantzng mages wth wods, Fst Intenatonal Wokshop on Multmeda Intellgent Stoage and Reteval Management, 999.