Symmetry Integrated Region-based Image Segmentation

Transcription

1 Smmetr Integrated Regon-based Image Segmentaton Yu Sun, Br Bhanu Center for Research n Intellgent Sstems, Unverst of Calforna at Rversde, Rversde, CA, 95 sun@ee.ucr.edu, bhanu@crs.ucr.edu Abstract Smmetr s an mportant cue for machne percepton that nvolves hgh-level knowledge of mage components. Unlke most of the prevous research that onl computes smmetr n an mage, ths paper ntegrates smmetr wth mage segmentaton to mprove the segmentaton performance. The smmetr ntegraton s used to optmze both the segmentaton and the smmetr of regons smultaneousl. Interestng ponts are ntall extracted from an mage and the are further refned for detectng smmetr axs. A smmetr affnt matrx s used explctl as a constrant n a regon growng algorthm n order to refne the smmetr of segmented regons. Expermental results and comparsons from a wde doman of mages ndcate a promsng mprovement b smmetr ntegrated mage segmentaton compared to other mage segmentaton methods that do not explot smmetr.. Introducton Snce most natural obects exhbt dfferent levels of smmetr, smmetr detecton has become a maor research topc n computer vson and pattern recognton. Research on mage smmetr can be manl dvded nto two schemes: local smmetr detecton [, ], and global smmetr detecton [3, 4]. The earler work on local smmetr detecton explored smmetr b local features such as edges, shapes, contours or boundar ponts. However, these methods can onl detect local smmetr propertes wthout consderng the whole mage. To solve ths problem, recent global smmetr detecton methods bascall treat the entre mage as a sgnal from whch smmetr propertes are dscovered. These approaches successfull detect smmetr nformaton from complex mage patterns n ether a dscrete or a contnuous smmetr detecton framework. Current global dscrete smmetr detecton approaches are revewed n []. More recentl, the use of smmetr as an mportant feature has been a topc wth sgnfcant attenton n new applcatons. The research has been done to appl smmetr nto the areas of pattern classfcaton [5] and obect recognton [6]. The smmetr feature along wth other features has been used as one of the ke multple cues. In ths paper, we use smmetr to mprove mage segmentaton, whch establshes a broad new doman of mage analss. The obectve of our work s to enforce smmetr constrants nto a regon-based mage segmentaton algorthm to mprove ts performance. Snce global smmetr detecton has advantages of freedom from a pror model, and robustness to complex patterns and dstortons, t s sutable for our regon-based mage segmentaton scheme. Moreover, dscrete smmetr detecton s necessar n order to enforce smmetr nto regon-based mage segmentaton methods such as regon growng, meanshft and watershed. Although these segmentaton methods var n how to obtan the regons, all of them have one thng n common the all defne a dscrete threshold settng, whch can be used to utlze the smmetr crteron that s also dscrete. Meanwhle, snce regon-based mage segmentaton accepts multple clusterng crtera, prevous work successfull ntegrated numerous crtera lke color, texture and shape pror. For our work, smmetr s combned as a new effcent crteron. The rest of ths paper s organzed as follows. In secton, we gve an overvew of related work and provde our contrbutons. In secton 3, the techncal approach of our work s proposed. Secton 4 provdes expermental results and dscussons. Fnall, conclusons are gven n secton 5.. Related Works and Our Contrbutons The approaches n [, ] manl detect smmetr b local shapes. In these papers, the local feature s the source of lmtatons, because of ts senstvt to pattern complext and dstorton. The ntuton of smmetr groupng s proposed n [5], whch s robust to more complex shapes. However, the are onl used n mages wth lmted pattern classes. Recentl, some researchers n [3, 4, 7] have taken a global approach to solve the above problems, where the nvestgate smmetr cues based on an entre mage. These knds of approaches, also used n our method, are relatvel more robust to complex and dstorted data /09/$ IEEE 86 Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.

2 The progress n smmetr detecton enables us to explot smmetr to other areas. However, onl a few papers are found n applcaton of smmetr to areas such as mage segmentaton. Two publcatons on smmetr enhanced mage segmentaton are found n [8, 9]. However, both of these papers have several lmtatons as gven below: Lmtaton : The combne smmetr wth onl a sngle segmentaton crteron, namel the edge crteron, such as actve contour [8] and edge weghts [9], that ams to fnd the contour of the smmetrc regon. The segment the mage nto onl two knds of regons: smmetrc regon nsde the contour and non-smmetrc regon outsde the contour, that do not segment the mage nto regons wth dfferent propertes such as color, texture and shape. Lmtaton : These methods use local features to segment smmetrc regons. Snce local features are senstve to non-smmetrc dstorton and nose, the have dffcultes wth the segmentaton of smmetrc regons wth ambguous, complex and nos patterns. And local features alwas fal to detect global domnant smmetr propert of the whole mage. As compared to the above lmtatons, our approach demonstrates the followng contrbutons:. Whereas most other research work stops at detectng smmetr, we go further to use smmetr to mprove mage segmentaton. Smmetr and segmentaton are mproved smultaneousl wth a satsfactor performance.. Our method solves the problem of lmtaton, b usng multple crtera (rather than a sngle crteron) ntegrated wth smmetr. As a result, regons wth dfferent propertes lke color, texture and smmetr are segmented smultaneousl. 3. Our method solves the problem of lmtaton, b usng global smmetr detecton, nstead of local smmetr detecton. Hence our approach s more robust to ambguous and complex non-smmetrc dstortons. 3. Techncal Approach The overall procedure s summarzed n Fg : Fg. Sstem dagram for smmetr ntegrated segmentaton. The ke procedures are smmetr ntegraton b smmetr affnt, and the mult-obectve optmzaton. The frst procedure realzes the smmetr constrant enforcement, whle the second procedure refnes the sstem through a closed-loop self-adustable optmzaton procedure. 3.. Dscrete Reflectve Smmetr Detecton We use Global Smmetrc Constellatons of Features n [7] to detect reflectve smmetr presentng n natural mages. Reflectve smmetrc pars are frst selected from SIFT ponts [3]. Pars of ponts whch pass ths selecton are then made to vote for domnant smmetr axs n Hough Space. A sample result s shown n Fg (b). Once the smmetr axs s detected, t s utlzed to compute a smmetr affnt matrx. Each pont n affnt matrx corresponds to a pxel n an mage, and ts value s the pxel s smmetr level as t relates to ts smmetrc counterparts. We emplo Curvature of Gradent Vector Flow (CGVF) [0], as a measurement of smmetr affnt: Curv v u uv u v v u V Let the GVF of mage be: In equaton (), u = u / x, u = u /, v = v / x, x v = v / are the frst dervatves of pxel along x and (x,) = [( ) ] 3 x + x V = [ u( x, ), v( x, )] drectons. Consderng a pxel ( x, ), we defne ts smmetr affnt as: C( x, ) = Curv( x, ) Curv( x, ) where ( x, ) s the smmetrc counterpart of ( x, ) b the smmetr axs. If the two ponts have locall smmetrc felds, then values of Curv( x, ) and Curv( x, ) should be closer. Three other smmetr condtons are stated n [0], whch can be combned wth (3) to buld the affnt matrx. 3.. Smmetr-based Regon Growng The regon growng segmentaton has the advantages of smplct, speed and the capablt of parallel clusterng, but has the lmtatons of over-segmentaton and s senstve to nose. Our method ams to solve these problems b ntegratng smmetr as a new constrant n regon growng segmentaton. The seeds for regon growng are chosen b the SIFT operator as n secton 3.. The smmetr affnt matrx s used to compute the smmetr constrant. Aggregaton Crteron: Image segmentaton concerns the partton of pxels nto regons wth unform propertes measured b homogenet crtera, lke color, texture, etc. x () () (3) 87 Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.

3 Let δ ( x, ) denotes the homogenet aggregaton crteron for a regon growng scheme. Ths crteron holds true when: δ (, ) < δm that means pxel wll be aggregated nto a neghborng regon f the homogenet crteron between them satsfes a threshold δ m. Naturall, common regon homogenet crtera used are color, texture and grascale. In ths paper, the aggregaton crteron s modfed so as to ntegrate the smmetr constrant. The modfed crteron s defned as: δ (, ) = δ (, ) δ (, ) R S where we enforce smmetr constrant δ s (, ) along wth regon crteron δ R(, ). Smmetr constrant δ s (, ) wll be ntroduced below. The regon homogenet crteron δ R (, ) s the combnaton of color and texture, that wll be provded n Secton 3.3. From ths work, a new crteron s realzed wth multple cues. Smmetr Constrants: δ s (, ) n equaton (5) s related to our new smmetr constrant, as shown below: π + actan( ( + C )( + C )) + C C δ s (, ) = + π C and C are smmetr affntes of pxel and neghborng regon, usng equaton (3). Equaton (6) provdes the followng smmetr constrants: the frst term controls the smmetr level, whch means that f both patterns and ndcate low smmetr affntes (hghl smmetrc), the are more lkel to be aggregated b decreasng the crteron δ (, s ) ; whle the second term favors more smlar smmetr affntes between the two patterns. Work n [9] uses smmetr crteron ntegrated nto edge weght n mage-cut segmentaton, and ts lmtatons are stated n secton. Regon Mergng Crteron: Intal segmentaton b the aggregaton crteron δ (, ) s an over-segmentaton result. Here the neghbored homogenous regons are merged b a measure of ther HSV color smlart b equaton (9). Accordng to both aggregaton and mergng crtera stated above, we establsh a D parameter search space used for segmentaton optmzaton, as ntroduced n Secton Color and Texture Aggregaton Crteron Our segmentaton procedure abdes both color and texture n the regon homogenet crteron δ R (, ), as gven b equaton (7). In partcular, the weghts of color and texture crtera can be adusted dnamcall durng the regon growng, dependng on whether the regon shows more unformt on color or more on texture. δ (, ) = W δ (, ) + W δ (, ) R Color Color Texture Texture (4) (5) (6) (7) Where W. We emplo HSV color space as the bass Texture + Wcolor = of color feature, that can be expressed as a 3D vector: F ( ) = ( v s cos( π h ), v s sn( π h ), v ) Color where h, s and v correspond to HSV components of pxel. The color homogenet crteron n equaton (7) can be expressed as: δ (, ) = F ( ) F ( ) whch s the Eucldean dstance of color features between pxel and ts neghborng regon. The 8-dmenson texture feature F Texture s derved from the mean and standard devaton of the fltered mage b Gabor flters at 4 orentatons b steps of 45 degrees. It s reduced to 4 dmensons b Prncpal Component Analss (PCA). Followng shows the texture homogenet crteron: Here both the color and texture features are normalzed Performance Evaluatons (8) (9) (0) Dfferent from the prevous work, our method mproves segmentaton and smmetr smultaneousl b smmetr ntegraton. Therefore, both segmentaton and smmetr evaluatons are requred as shown n the followng: S E G K R e S E G ( ) = ( + lo g ( R ) ) [ ] M * N + lo g ( A ) SYM K = e R ' SYM (, ) N R = () () For segmentaton evaluaton of equaton (), M*N s the sze of mage, and R s the number of segments. The term e SEG ( ) s the nter-regon contrast of regon : e Color Color Color δ (, ) = F ( ) F ( ) Texture Texture Texture SEG ( ) = N = F ( ) F Color ( R ) Color N (3) FColor ( ) FColor ( R ) s the pxel contrast n the regon, whch s the Eucldean dstance of color features between pxel and ts regon R. N s the number of pxels n regon. The term ( + log( R)) s a punshment for over-segmentaton, and ( + log( A )) s a punshment for small segments, and A s the area of regon. For smmetr evaluaton of equaton (), ' esym (, ) s the dfference of regon propertes between regon and ts smmetrc counterpart b the smmetr axs. The regon dfference s denoted b the sum of ther ' mean color and orentaton dfference. The smaller esym (, ) means more smmetrc the regon s to ts smmetrc counter part. Smaller value of evaluaton ndcates better performance. For convenence purpose, both evaluatons are adusted so that larger s the better n experments. = 88 Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.

4 3.5. Mult-obectve Optmzaton The purpose of the optmzaton (see Fg ) s to get a segmentaton result L(x) wth both ts segmentaton and smmetr performances optmzed, that can be formulated as a mult-obectve optmzaton problem. We use Non-domnated Sortng Genetc Algorthm (NSGA-II), a mult-obectve optmzaton approach used n [] to conduct the search for optmum combnaton of parameters, b means of non-domnated sortng of a combned parent and offsprng populaton, whch prevent both the local optma and lack of robustness. The optmzed results from dfferent segmentaton methods are shown n Fg Smmetr Integraton Algorthm The sstem s summarzed as the followng algorthm: Global Smmetr Detecton. Smmetr pars extracted from SIFT nterestng ponts;. Compute smmetr affnt b Curvature of GVF; Smmetr-ntegrated Regon Growng Segmentaton 3. Seeds determnaton b nterestng ponts; 4. Compute multple aggregaton crtera for regon growng: color crteron b HSV bass, texture crteron b Gabor flters, and smmetr crteron b smmetr affnt. Combne them as a sngle aggregaton crteron, wth ts threshold AGC (see Secton 4.); 5. Run regon growng; 6. Use HSV color bass to compute regon mergng crteron, wth ts threshold MGC (see Secton 4.); 7. Run regon mergng and fnsh the segmentaton; Segmentaton Optmzaton 8. Segmentaton and smmetr performance evaluatons; 9. Mult-obectve optmzaton b NSGA-II, n parameter space of AGC and MGC, s run b the followng rule: If segmentaton and smmetr performances from 8 are acceptable (see Secton 4.), optmzaton s ended; If not acceptable, search dfferent parameter settng of AGC and MGC, and ump to Expermental Results and Dscussons In ths secton, the performance of our method, named Smmetr-ntegrated Regon Growng, s compared wth four segmentaton approaches: color & texture regon growng, gra-scale regon growng, meanshft and watershed. Our method provdes superor segmentaton performance meanwhle the smmetr s hghl preserved. 4.. Datasets and Parameters The proposed method s appled to mages demonstratng a wde range of regon propertes, as llustrated n Fg 3. The parameter space for segmentaton and smmetr optmzaton s composed of thresholds, wth respect to multple aggregaton crtera (AGC) threshold and regon mergng crteron (MGC) threshold, whch buld a D parameter space as n Fg 4(b). The mult-obectve optmzaton stops f both the performances of segmentaton and smmetr are acceptable as follows: a. Both segmentaton and smmetr performances are better than pre-set thresholds (set b 0.6 and 0.70, respectvel); b. The combnaton of the two performances reaches ts global optmum value n NSGA-II. The optmzaton stops f both condtons are met, otherwse t wll contnue b searchng the dfferent combnaton of parameters recursvel. 4.. Smmetr-constraned Regon Growng The results of smmetr-based regon growng are presented n Fg. Fg (a) shows the orgnal mage for a smmetrc butterfl bod wth complex non-smmetrc, and Fg (b)-(d) show the smmetr axs, the segmentaton result, and the ground-truth segmentaton respectvel. Fg (e) shows the curve of relatonshp between smmetr and segmentaton performances. Each pont n the curve s obtaned b runnng segmentaton b a parameter settng wthn the parameter space, and gettng evaluatons of segmentaton and smmetr, both of whch can be mproved smultaneousl as ndcated b the curve. Fg (f) shows the recever operatng characterstc (ROC) plot, whch ndcates the classfcaton accurac of the result n Fg (c) compared to the ground-truth segmentaton n Fg (d). The plot s derved from statstcal results b runnng segmentaton n wde range of parameter space Segmentaton Improvement b Smmetr We get a decent segmentaton mprovement usng smmetr constrants when compared to other segmentaton methods wthout explotng smmetr. All the results n Fg 3(a)-(d) are optmzed results b NSGA-II. We nfer from Fg 3 that the most complete and complex smmetrc obects are segmented b our proposed method as shown n results n (a), as compared to (b)-(d), whch have segmentaton defects n smmetrc regons. Once regons ndcate smmetrc propert, the are more lkel to be aggregated b smmetr constrant. As an example, for the result (a) of mage (4) n Fg 3, our approach can segment the whole smmetrc face shape, regardless of ts nose and holes, whle other results fal to accomplsh so. The percentage data n Table represents the segmentaton mprovement compared to the result of the method n the next row, wth hghest performance from smmetr-ntegrated method. Fg 4(a) shows the curves between smmetr and segmentaton performances for fve segmentaton methods. The squared ponts marked as a - e are the mult-obectve optmzaton results b NSGA-II. A comparson among fve curves 89 Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.

5 (a) Orgnal mage (b) Smmetr axs & ponts (c) smmetr-based segmentaton (d) Ground-truth (e) Performances: smmetr vs. segmentaton (f) ROC plot between (c) and ground-truth (d) Fg.. Smmetr-constraned segmentaton results and analss. Image () Butterfl () Butterfl Regon propertes Unform smmetrc regons wthout Textured smmetrc regons wthout (3) Butterfl 3 (4) human Unform & textured Unform & textured smmetrc regons wth smmetrc regons wth (5) Car Unform smmetrc regons wth Orgnal GroundTruth (a)smmetr based Color & Texture Regon Growng (b) Color & Texture Regon Growng (c) Meanshft (d) Gra-scale Regon Growng Fg. 3 Sample segmentaton results: sample mages ()-(5) wth results (a)-(d) from 4 segmentaton methods. 830 Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.

6 ndcates that the smmetr-ntegrated segmentaton of curve a acheves the best performances. Moreover, Fg 4(a) ndcates that the performance mprovement of curve a compared to curve b comes onl from smmetr ntegraton, that both curves a and b use the same color-texture regon growng approach, but the method of curve a ntegrates smmetr as a new constrant. Fg 4 (b) shows the clusters of both segmentaton and smmetr performances on the D parameter space, where parameter means the regon aggregaton crteron (AGC) threshold, and parameters ndcates the regon mergng crteron (MGC) threshold. The results ndcate that b applng smmetr constrants, both segmentaton and smmetr are mproved n most part of the parameter space. Table.. Smmetr/segmentaton evaluaton (segmentaton performance mprovement), compared to the method of next row. () Butterfl () Butterfl (3) Butterfl 3 (4) human (5) Car (a) Smmetr based RG 0.85 / (7.5%) 0.78 / (6.6%) 0.66 / (3.6%) 0.668/ 0.79(6.4%) / (4.8%) (b) Color & Texture RG / 0.83 (.%) / 0.95 (4.0%) / 0.9(6.3%) / (6.%) 0.63 / (.5%) (c) Meanshft / (7.7%) / (.3%) / 0.79(9.5%) / (7.9%) / 0.60 (9.6%) (d) Regular RG 0.83 / 0.739(0.8%) 0.78 / (.4%) / 0.60(6.%) / (3.%) / (8.5%) (e) Watershed / / / / / 0.5 Fg 4. (a) Smmetr and segmentaton performance (b) segmentaton and smmetr performances of mage butterfl 3 curves for mage butterfl 3. n large parameter space. 5. Conclusons In ths paper, a new smmetr ntegrated regon-based segmentaton scheme s proposed for natural mage segmentaton. We perform experments on a wde varet of mages. The qualtatve and quanttatve expermental results ndcate that wth the smmetr constrants enforced b smmetr affnt, both the smmetr and segmentaton are mproved, wth better performance compared to several other well known regon-based segmentaton methods. Acknowledgement: Ths research was supported b NSF grant The authors would lke to thank Dan Janzen for the butterfl mages and John Herat for the comments. References [] D Ravv, A. M. Bronsten, M. M. Bronsten, R. Kmmel. Smmetres of Non-rgd Shapes. ICCV, 007. [] S. Thrun, B. Wegbret. Shape from Smmetr. ICCV, 005. [3] J. Stahl, S. Wang. Globall Optmal Groupng for Smmetrc Boundares. CVPR, 006. [4] H. Cornelus, G. Lo. Detectng Blateral Smmetr n Persp- ectve. CVPRW, 006. [5] L. Lucchese. A Frequenc Doman Algorthm for Detecton and Classfcaton of Cclc and Dhedral Smmetres n Two-dmensonal Patterns. ICIP, 00. [6] T. Lu, N. Zheng, L. Zhao, H. Cheng. Learnng Based Smmetrc Features Selecton for Vehcle Detecton. IEEE Proc., Intellgent Vehcles Smposum, 005. [7] G. Lo, J. Eklundh. Detectng Smmetr and Smmetrc Constellatons of Features. ECCV, 006. [8] T. Rkln-Ravv, N. Krat, N. Sochen. Segmentaton b Level Sets and Smmetr. CVPR, 006. [9] A. Gupta, V. S. N. Prasad, L. S. Davs. Extractng Regons of Smmetr. ICIP, 005. [0] V. S. N. Prasad, B. Yegnanaraana. Fndng Axes of Smmetr From Potental Felds. IEEE Trans. on Image Processng, Vol. 3, No., 004. [] M. Park, S. Lee, P. Chen, S. Kashap, A. A. Butt, Y. Lu. Performance Evaluaton of State-of-the-Art Dscrete Smmetr Detecton Algorthms. CVPR, 008. [] S. P. Kodal, R. Kudkala, Deb K. Mult-Obectve Optmzaton of Surface Grndng Process Usng NSGA-II. ICETET, 008. [3] D. G. Lowe. Dstnctve Image Features from Scale-Invarant Keponts. Internatonal Journal of Computer Vson, Authorzed lcensed use lmted to: Unv of Calf Rversde. Downloaded on September, 009 at 3:4 from IEEE Xplore. Restrctons appl.