A Semantic Region Growing Approach in Image Segmentation and Annotation

Size: px

Start display at page:

Download "A Semantic Region Growing Approach in Image Segmentation and Annotation"

Stephany Wheeler
5 years ago
Views:

1 A Semantc Reon Grown Approach n Imae Sementaton and Annotaton Thanos Athanasads, Yanns Avrths and Stefanos Kollas Imae, Vdeo and Multmeda Systems Laboratory School of Electrcal and Computer Enneern Natonal Techncal Unversty of Athens 9, Iroon Polytechnou St., Zoraphou, Greece {thanos,avr}@mae.ntua.r, stefanos@cs.ntua.r Abstract. In ths poston paper we examne the lmtaton of reon rown sementaton technues to extract semantcally meannful objects from an mae. We propose a reon rown alorthm that performs on a semantc level, drven by the nowlede of what each reon represents at every teraton step of the mern process. Ths approach utlzes smultaneous sementaton and labeln of reons leadn to automatc mae annotaton. 1. Introducton Automatc sementaton of maes s a very challenn tas n computer vson and one of the most crucal steps toward mae understandn. A varety of applcatons such as object reconton, mae annotaton, mae codn and mae ndexn, utlze at some pont a sementaton alorthm and ther performance depends hhly on the ualty of the latter. It s acnowleded that aes-lon research has produced alorthms for automatc mae [1] and vdeo [2] sementaton, structurn of multmeda content [3] and reconton of low-level features wthn such content [4]. Comparatvely to ths effort, lttle proress has been made on machne-enerated semantc descrptons of audovsual nformaton n a way famlar to humans. Stll, human vson percepton outperforms state-of-the-art computer s sementaton alorthms. The man reason for ths s that human vson s based also n hh level pror nowlede about the semantc meann of the objects that compose the mae. We propose a sementaton technue that belons to the eneral framewor of reon rown sementaton alorthms [5],[2]. Reon rown alorthms start from an ntal partton of the mae and then an teraton of reon mern bens, based on certan smlarty crtera untl the predefned termnaton crtera are met. Our contrbuton s an addtonal mern process that n comparson to prevous mern, ts crtera are not based on syntactc features le color or texture smlarty, but on matchn of concepts assocated to each reon. In other words, after a certan pont where syntactc reon mern stops, an ntal reon labeln s carred out usn low-level features and detectors [6] and then sementaton contnues based ths tme

2 on fuzzy crtera that apply on a semantc level,.e. the assned concepts to each reon alon wth a correspondn confdence value. 2. Semantc Reon Grown Alorthm The taret of ths novel alorthm s to mprove both sementaton and reconton of objects at the same tme, wth obvous benefts for semantc annotaton of maes. In the follown two subsectons we descrbe the foundatons of the Semantc Reon Grown (SRG) alorthm, whch are the raph representaton of the maes and the ntal selecton of the seeds. Fnally the proposed alorthm s examned n subsecton Graph Representaton of an Imae An mae can be descrbed as a structured set of ndvdual objects, allown thus a strahtforward mappn to a raph structure. In ths fashon, many mae analyss problems can be consdered as raph theory problems, nhertn the sold theoretcal rounds of the latter. Attrbuted Relaton Graphs (ARGs) are a type of raph often used n computer vson and mae analyss for the representaton of structured objects. In ths wor we adopt the formal representaton of an ARG ven by Berret et al n [7], where an ARG s defned precsely by spatal enttes represented as a set of vertces E, each labeled wth an attrbute a and, bnary spatal relatonshps represented as pars of vertces E E each labeled wth a spatal descrptor w. In partcular, the vertex s attrbute a s a complex structure that contans the follown two (also complex) enttes: 1. Three MPEG-7 Vsual Descrptors that descrbe the low-level features of the correspondn reon, namely Domnant Color, Reon Shape and Homoeneous Texture. 2. A lst of canddate labels, alon wth a deree of confdence for each one. Ths s the result of the ntal reon labeln, dscussed brefly n the follown secton. The spatal descrptor w contans nformaton reardn the spatal relaton of the reons, whch are actually extracted but not utlzed so far from the alorthm under dscusson, remann hence an open ssue for future research. 2.2 Intalzaton of Reon Labeln Our ntenton s to wor on a hher level of nformaton where reons are lned to possble labels rather than only to ther vsual features. The above descrbed ARG contans low-level nformaton extracted drectly by the mae tself, but t also has labels and confdence values assned by a nowlede-asssted analyss (KAA) alorthm, dscussed n depth n a prevous wor [6]. For each vertex (.e. a reon of the mae) of the ARG a matchn process s performed between the vsual descrptors stored n the vertex and the correspondn vsual descrptors of concepts, stored n the

3 form of prototype nstances n an ontolocal nowlede base. Ths process results to an ntal fuzzy labeln of the reons wth concepts from the nowlede base. Ths s of course not a smple tas and results depend hhly on the doman where t s appled, as well as on the ualty of the nowlede base. 2.3 SRG Alorthm Descrpton Conductn thorouh experments tryn to mprove the results of the KAA alorthm, we came up wth the dea presented n ths paper: To adapt a well nown sementaton technue, le reon rown, to the problem of semantc annotaton. More specfcally, we adopt a watershed-le reon mern [8] technue, startn from reonsseeds that are automatcally selected. Let us now ntroduce the necessary mathematcal notaton used n ths paper. The Semantc Reon Grown (SRG) alorthm acts on a hher level than other reon rown alorthms; ths hher level we call t Semantc Level: { } SL = L, D, RG The fuzzy set SL that represents the Semantc Level, conssts of pars L, D, for all reons of the mae: RG, where RG :the set of all reons n the mae. The enttes L and D are two sets contann for the specfc reon all canddate labels and confdence values respectvely and, are defned as: L = { l} L, where: RG, l L, (2) where L :the set of all possble labels (1) D = { dl }, where d [0,1] l (3) The aforementoned par formulates that every snle reon has been assned to a number of canddate labels (euaton 2) accompaned by the respectve confdence values (euaton 3): RG:has a set of labels L = { l } L l : has a confdence value d l (4) A number of reons are selected to be used as seeds for the ntalzaton of the SRG alorthm and form an ntal set, let t be S. The crtera for selectn a reon to become a seed are two: ) The reon s best confdence value should be above a threshold. ) the rest concepts have low confdence values. These two constrans en-

4 sure that the specfc reon has been correctly selected as seed of the partcular concept. An teratve process bens that checs whether the drect nehbors (as defned n the ARG) of the ntal reons-seeds have been assned to the same concept ts propaator reon-seed has and, wth what confdence value. Some of those reons, that satsfy two addtonal crtera, form a new set of reons N ( denotes the teraton step, wth S ), whch wll be the new seeds for the next teraton of the alo- 0 rthm. These two crtera are: 1. Confdence value of the propaator reon p for the partcular label l should be above a threshold: d p l > T prop 2. Confdence value of the reon under examnaton for the same label l should be above another threshold: dl a T chld >, where a s a constant slhtly above one, that ncreases the threshold n every teraton of the alorthm n a nonproportonal way to the dstance from the ntal reons-seeds. When the above crtera are satsfed, reon s mered wth ts propaator and ts confdence value s re-evaluated as the mnmum between ther confdence ˆ p values, thus: dl = mn ( d, ) l d l The termnaton crtera of the alorthm are ute strahtforward: Repeat whle the set of reons-seeds n step : N. In ths pont, we should underlne that when nehbors of a reon are examned, prevous accessed reons are excluded,.e. each reon s reached only once and that s by closest reon-seed, as defned n the ARG. Schematcally, ths alorthm loos le clusters of reons (each cluster correspondn to a specfc concept) expandn n every teraton, untl ether the coherency of the cluster s smaller than allowed to be, or the borders of two such clusters meet. We use the term watershed-le because the decson for whch reons to be mered depends on both ther confdence value and ther dstance from the seed (catchment basn, n watershed sementaton termnoloy) and the teraton eeps on untl two expanded reons meet (basns are flooded tll the watershed). p 3. References 1. H. Gao, W.-C. Su and C.-H. Hou, Improved technues for automatc mae sementaton, IEEE Trans. on Crcuts and Systems for Vdeo Technoloy, vol.11, no. 12, pp , December P. Salember, F. Marues, Reon-Based Representatons of Imae and Vdeo - Sementaton Tools for Multmeda Servces, IEEE Trans. on Crcuts and Systems for Vdeo Technoloy, vol.9, no.8, December S. Chan and H. Sundaram, Structural and Semantc Analyss of Vdeo, IEEE Internatonal Conference on Multmeda and Expo (II), E.L.Andrade Neto, J.C.Woods, E.Khan, M.Ghanbar Reon Based Analyss and Retreval for Tracn of Semantc Objects and Provson of Aumented Informa-

5 ton n Interactve Sport Scenes, IEEE Trans. on Multmeda Vol. 7, Issue 6, Dec Pae(s): R. Adams and L. Bschof, Seeded Reon Grown, IEEE Trans. on Pattern Analyss and Machne Intellence, vol 16, no. 6, pp , June T. Athanasads, V. Tzouvaras, K. Petrds, F. Precoso, Y. Avrths and Y. Kompatsars, Usn a Multmeda Ontoloy Infrastructure for Semantc Annotaton of Multmeda Content, Proc. of 5th Internatonal Worshop on Knowlede Marup and Semantc Annotaton (SemAnnot '05) 7. S. Berrett, A. Del Bmbo, E. Vcaro, Effcent matchn and ndexn of raph models n content-based retreval, IEEE Trans. on Crcuts and Systems for Vdeo Technoloy, vol.11, no. 12, pp , December S. Beucher and F. Meyer, The Morpholocal Approach to Sementaton: The Watershed Transformaton, n: Mathematcal Morpholoy n Imae Processn, E.R.Douhertty (Ed.), Marcel Deer, NY, 1993.

Affine Invariant Matching of Broken Boundaries Based on Differential Evolution

Affine Invariant Matching of Broken Boundaries Based on Differential Evolution Affne Invarant Matchn of Broken Boundares Based on Dfferental Evoluton Wuchao tu P.W.M. Tsan Department of Electronc Enneern Cty Unversty of Hon Kon Hon Kon Chna Abstract - Affne nvarant matchn of a par