Image Segmentation with Perceptual Guidance

Transcription

1 Image Segmentaton wth Perceptual Gudance Xaofeng M Department of Computer Scence Rutgers Unversty xm@cs.rutgers.edu Fgure 1. Left top: the orgnal Davd head mage, left bottom, segmented mage wth mean shft on L*a*b* space powered wth edge confdence gudance, Rght: segmented mage wth mean shft on L*a*b* space plus perceptual measure Abstract Ths paper proposes an alternatve way of embeddng confdence nformaton nto the segmentaton procedure. Two dfferent types of percepton measures and how to evaluate the measures are descrbed. Instead of usng these percepton measures for pxel weght durng segmentaton, the proposed approach smply adds the percepton measure as an addtonal feature space dmenson. Takng human percepton nformaton nto account makes the segmentaton results match human vsual percepton more accordngly and thus are more consstent wth actually human segmented results lke rotoscopng. Experment results show that ths approach produces aesthetc pleasng segmentaton results, whch s useful for further mage based depcton. 1 Introducton Image segmentaton, whch ams to partton the mage nto semantcally meanngful regons, s arguably consdered as the most mportant low level vson operatons. Durng recent years, mage segmentaton has found ts applcatons n computer graphcs researches. Barrett and Cheney proposed an nteractve mage edtng system based on mage segmentaton [Barrett 0]. DeCarlo and Santella proposed an mage abstracton technque by segmentng mages on the Gaussan pyramds then selectng approprate segmentaton scales accordng to human nteracton data [DeCarlo 0]. Zhou et al advocated the use of a hybrd focal regon-based volume renderer that offers an alternatve for vsualzaton of nternal structures of medcal magng data [Zhou 0], where segmentaton nformaton was ntroduced to enhanced renderng. Ths technque s also appled recently n vdeo segmentaton to produce hghly abstracted, cartoon style vdeo wth temporary coherence [Wang 04] However, the result drectly from mage segmentaton s usually undesrable for applcatons n renderng. One of the bggest problems s that the segmentaton boundares are too arbtrary. Also, segmentaton scales are usually not determned automatcally. For example, n Fgure 1, the mage at left bottom s the segmented result from the orgnal mage at left top, drectly usng mean shft segmentaton on L*a*b* space wth careful chosen bandwdths such that mportant features lke human eyes get kept. However, to keep such features as further renderng nformaton wll result n a sgnfcant clutter appears. Actually, wthout human nteracton, n the tradtonal segmentaton approach n Eucldan L*a*b* 1

2 a b Fgure. a s from sprted away by Myazak from Ghbl studo, whose style tends to use two colors, lght color and shadow color, to llustrate characters. In rotoscoped mages lke the one shown n b, the artsts partton the area nto varous knd of part whle keepng the partton meanngful n some sense. space, keepng detals and keepng from non-salent regons are always a par of conflct. On the other hand, from an aesthetcal vewpont, stylzaton s a most mportant ssue and the key problem n a non-photorealst renderng system s to drect these resources of style to preserve meanngful vsual form, whle reducng extraneous detal. Vsual form descrbes the relatonshp between pctures of objects and the physcal objects themselves. A good example s the cartoon style abstracton, where human faces are depcted n very lmted knds of colors and the boundares between mage segments are determned unformly for example, gven a partcular cartoon style, most of the artsts wll follow a common way to place the regon boundares to depct human faces accordng to the shapes and the colors. Fgure shows some pctures from actual moves. In the left mage, the chld s face s rendered n a much unform color, as s the case wth the pg (her parents accordng to the gut though). Whle n the rght, whch was taken from the famous move wakng lfe and was produced based on human rotoscopng, s a knd of more pantng-style abstracton. Segmentng mages drectly from pxel color nformaton s not a good approach as shown n fgure 1. And a lot of researches have been focused on combnng the outputs of mage segmentaton and edge detecton to mprove the qualty of the segmented mage. Frexenet et al surveyed seven dfferent strateges to combng smlarty (regon) and dscontnuty (edge) nformaton, n ether embedded or post-processed approach [Frexenet 0]. However, as s show n fgure 3(b), such knd of approach stll suffers from the dlemma between scale selecton and gettng rd of nonsense regon parttons. Ths paper proposes an alternatve method of embeddng confdence nformaton nto the segmentaton procedure. Instead of usng these percepton measures for pxel weght, ths paper smply adds the percepton measure nto the feature space as an addtonal channel. Ths s a b Fgure 3. a s the orgnal mage and b s the segmentaton result by EDISON [Chrstoudas 0] wth spatal bandwdth 7 pxels and color bandwdth (unformly n L*u*v* space) 7, wth synergetc embedded edge confdence. As show n b, although the area around eyes begns to get rough, there are stll a lot of undesred regon parttons.

3 based on the observaton that human percepton has some sorts of contnuty on a partcular feature, and ths contnuty wll also gude human s vsual percepton. That s, human wll feel natural f an mage s segmented n accordance wth ther percepton to the world. That also explans why artsts wth the same artstc abstracton style wll segment a gven mage n a roughly same way. Mean Shft Image Segmentaton A large class of current mage segmentaton algorthms are based on feature space, by analyzng the mages n some carefully defned space typcally color spaces or texture spaces. The later are of partcular sgnfcance n texture segmentaton. In segmentaton ths way, mage segmentaton s smply a partton of an mage nto contguous regons of pxels that have smlar appearance, such as color or texture [Trucco and Verr 98]. Each regon has aggregate propertes assocated wth t, such as ts average color. The algorthm descrbed by Comancu and Meer [Comancu 0] s a paradgm of robust segmentaton of color mages, as t produces relatvely cleaner results despte of some sort of mage noses. Wthn ths algorthm, colors are represented n the perceptually unform color space L*u*v* [Foley et al. 97] whch produces regon boundares that are more meanngful for human observers. The parameters of ths algorthm nclude a spatal radus h s (smlar to the radus of a flter), a color dfference threshold h r, and the sze of the mnmum acceptable regon M. The mappng of real mages to feature spaces often produces a very complex structure. Salent features whose recovery s necessary for the soluton of the segmentaton task, correspond to clusters n ths space. As there s no a pror nformaton s typcally avalable, the number of clusters/classes and ther shapes/dstrbutons have to be dscerned from the gven mage data. Let {X } =1 N be the set of N mage vectors n the d-dmensonal Eucldean space R d. The multvarate kernel densty estmate obtaned wth kernel K(x) and wndow radus h (the unversal bandwdth here), computed at pont x s defned as the well known expresson: N 1 x X fx ˆ( ) = d K ( Nh ) (1) = 1 h The radally symmetrc kernel functon K(x) satsfes: Kx ( ) = c ( kd, k x ) () then, we rewrte the kernel densty estmator as: N ˆ ckd, x X, ( ) fhk x = k d (3) Nh = 1 h The modes of the densty are located among the zeros of the gradent f () x = 0. Defne: gx ( ) = k'( x), Gx ( ) = c ( gd, g x ) (4) We have f () x = fˆ () x m () x hk, hg, hg, where f ˆ () hg, x s proportonal to the densty estmate at x computed wth the kernel G, and N x x xg 1 = h mhg, () x = x N x x (6) g = 1 h s known as the mean shft, whch shows that the estmaton densty gradent at locaton x s proportonal to the offset of the mean vector computed n a wndow, and recursve applcaton of the mean shft property make the pxels converge to ther local densty modes and thus pxels can be parttoned wth local modes. Ths s the bass of mean shft segmentaton. 3 Vsual Percepton In tradtonal color space based segmentaton, L*a*b* or L*u*v* color spaces are regarded as the best sutable for vson research because they have a metrc a satsfactory approxmaton to Eucldean, thus allowng the use of sphercal wndows. Though human percepton towards an mage s determned by the colors of the pxels, the percepton of each pxel s not determned solely wth that pxel. For example, whle a lot of color space model have an explct or mplct representaton of color lumnance, such as L*a*b*, L*u*v*, HSV and grayscale color space as well, human does not lkely percept brghtness just accordng to local vson area, nstead, the surroundng vsual area has sgnfcant mpact on the human perceptons and human tends to dscard certan propertes of lght, based on the prncple, comes the frst percepton measure used as the mage feature space for segmentaton. 3.1 Brghtness Measure As s surveyed by Gooch et al[gooch 04], brghtness percepton can be modeled usng operators such as dfferentaton, ntegraton and thresholdng. These methods model lateral nhbton whch s one of the most pervasve structures n the vsual nervous system [Palmer 1999]. Lateral nhbton s mplemented by a cell s receptve feld havng a center-surround organzaton. Thus cells n the earlest stages of human vson respond most vgorously to a pattern of lght whch s brght n the center of the cell s receptve feld and dark n the surround, or vce-versa. Such antagonstc center-surround behavor can be modeled usng neural networks, or by computatonal models such as Dfference of Gaussans, Gaussan smoothed Laplacans and Gabor flters. (5) 3

4 Lke what Gooch et al dd n perceptual measure based human face llustraton [Gooch 04], ths paper follows Blommaert and Martens [Blommaert 1990] model of human brghtness percepton. The am of the Blommaert model s to understand brghtness percepton n terms of cell propertes and neural structures. For example, the scale nvarance property of the human vsual system can be modeled by assumng that the outsde world s nterpreted at dfferent levels of resoluton, controlled by varyng receptve feld szes. Blommaert and Martens demonstrated that, to a frst approxmaton, the receptve felds of the human vsual system are sotropc wth respect to brghtness percepton, and can be modeled by crcularly symmetrc Gaussan profles R : x + y ( αs) 1 R (,,) x y s = e πα ( s) (7) where s s the number of pxels nvolved n the correspondng level of brghtness recepton feld. α 1 = 1/( ) and α + 1 = 1.6 * α. The neural response V as the functon of mage locaton, scale and lumnance dstrbuton L can be computed by convoluton: V(,, x y s) = L(, x y) R(,, x y s) (8) The frng frequency evoked across scales by a lumnance dstrbuton L s modeled by a center-surround mechansm: V(,, xys) V (,, xys) V (,, x y s ) = / (,, ) + 1 φ s + V x y s φ where V s specfed by (8) and the term /s s ntroduced to avod sngular cases when V s approachng zero. And n the magng applcaton, φ = 1 s an approprate value. An expresson of brghtness B s now dervng by summng V over all scales: s max (9) B = V (, x y, s ) (10) s = s 0 Practcally, choosng the total number of scales up to 8 s Fgure 4. brghtness measure of fgure 3(a) actually good enough. The result of these computatons s an mage whch could be seen as an nterpretaton of human brghtness percepton. For example, the brghtness percepton of fgure 3(a) s shown as fgure 4, wth low percepton value shown n black and hgh value n whte. Addng the calculated brghtness mage nto the orgnal L*a*b* color space can help gudng mage segmentaton gven that approprate bandwdth at ths dmenson s chosen. The mage shown on the rght of fgure 1 s such an example and fgure 5 shows yet another example, whch s the segmented result for fgure 3(a). Fgure 5. Segmentaton result of mage shown n fgure 3(a) wth color space bandwdths and percepton bandwdth 0.1. (both color channel and percepton values take doman n [0, 1]) 3. Photometrc Invarant Measure As shown n fgure (a), n a broad range of applcatons, one tends to omt the surface orentaton and the shadng, hghlght or shadow effects and treats the whole object as a sngle segment, whch makes t obvous that photometrc nvarance s essental for mage segmentaton. Ths subsecton, wth the example, argues that by addng the hue value as the measure of photometrc nvarance, photometrc nvarant segmentaton could be retreved. An N-dmensonal spectrum can be denoted as: c= mb( nsc, ) b + ms( nsvc,, ) s (11) Where n, s, v are three-dmensonal, denotng the surface normal, the drecton of the llumnaton source and the drecton of the vewer respectvely and where c and c s are the surface albedo and Fresnel reflectance respectvely, whch are both N-dmensonal wth N the number of samples taken n the wavelength range. For example, n RGB color space, N wll be 3. Gevers argues that ths color space can be transformed nto a dfferent polar coordnate representaton as follows[gevers 03]: 4

5 ps = 1 mn{ c( λ1 ),... c( λn)} (1) θh = α[( c λ) [1 ρs], φ(, N)] (13) Where θh takes on values n the range 0 θh π and where 1 4 φ(, N) = π N 1 3 (14) and N w sn 1 θ α( w, θ) arctan = = N w cos 1 θ = (15) For RGB spectrums, we have 4 ( R ρs)sn(0) + ( G ρs)sn( π) + ( B ρs)sn( π) θ = arctan ( R ρs)cos(0) + ( G ρs)cos( π) + ( B ρs)cos( π) G B 3( G B) = arctan = arctan 1 1 ( R G) + ( R B) R G B (16) whch, by the defnton of hue value of color gven by Levkowtz and Herman [Levkowtz 93], shows that the value of θ s exactly the hue value of the color. As the brghtness percepton measure, addng the hue value as the percepton measure to the feature space wll help to produce photometrc nvarant segmentaton result as long as the approprate bandwdth s selected. Fgure a c Fgure 6. a shows the photometrc nvarant segmentaton result, b s the bnary mage for brghtness measure and c s the combnaton of the two mages. b 6(a) shows such the results for fgure 3(a). Interestngly, f we combne the bnary mage of the brghtness mage computed as shown n fgure 4 wth a brghtness threshold (shown n fgure 6(b)) and the photometrc nvarant segmentaton result, we can get a popular cartoon stylzaton result, as shown n fgure 6(c). Another example of the same knd s shown n fgure 7(d). 4 Dscusson and Concluson Ths paper proposed a paradgm of addng perceptual measures nto the feature space whle performng segmentaton to get more desrable results. Two perceptual measures are ntroduced and experments shows that ths method can help produce segmentaton results whch make much more sense from the perceptual and aesthetcal pont of vew. Such methods also provde an alternatve for scale control whle segmentaton. Compare the result of fgure 3 and fgure 5, t s obvous that by takng brghtness measure nto account, fgure 5 deals wth the scale near subtle areas lke eyes and others lke face area more sophstcally. The work descrbed n ths paper falls nto the paradgm of applyng vson technques n computer graphcs area. More examples wth the proposed approach are shown n Fgure 7 and fgure 8 There are open problems reman n addng addtonal feature dmenson though. One practcal ssue to deal wth s the bandwdth determnaton of perceptual feature space. Snce the measure of human percepton s totally another story other than color space, the bandwdth selecton at perceptual dmenson s currently done nteractvely. How to select a proper bandwdth for the perceptual measure stll needs more efforts. In fact, sometmes the bandwdths are not easy to choose. For example, to produce the mage shown n fgure 7(c), where the yellow ball on the torch beng a separate regon other than the sky or the green statue, the bandwdth les n a small range of the feature space and t really requres one wth great patence, whch s not a good practce (As for pure color-feature based mean shft, as the conflct between fne scale for detal and course scale for abstracton, the system can not produce a smlar result wth both the yellow ball preserved and the shadng effects elmnated). The examples shown throughout ths paper s segmented va mean shft approach. However, theoretcally, any feature-space based segmentaton should be applcable for the space wth perceptual measure added as addtonal dmenson. Both the brghtness and the hue percepton measure have lmtatons due to the sngular cases. As for brghtness measurement, snce t operates based on the lumnance level of pxels, t mght fall nto wrong way when the lumnance levels of pxels are smlar whle the colors are 5

6 dfferent. Though ths happens rarely, the problem wth hue ntegraton descrbed n secton 3. for photometrc nvarant segmentaton occurs much common: t suffers from the nstablty of the hue value around regons where R, G, B values are smlar. How to fnd a robust photometrc nvarant measure for dgtal mages s stll an open ssue. The approach descrbed n ths paper shows specal advantages when segmentng human faces usng brghtness perceptual measure, as can be seen from the examples. Whle the effect on the segmentaton of other type of objects remans a lttle bt controversal and need further exploraton. Acknowledgements Thanks to Prof. Elgammal and Prof. DeCarlo and Anthony Santella for the dscussons of the work whch have benefted me a lot. The mplementaton of mean shft segmentaton was largely borrowed from EDISON system. References [1] Wllam A. Barrett, Alan S. Cheney, Object-based mage edtng, Proceddng of SIGGRAPH 00, Pages: [] F. J. J. Blommaert, and J.-B. Martens1990. An object-orented model for brghtness percepton. Spatal Vs. 5, 1, [3] Chrstopher M. Chrstoudas, Bogdan Georgescu, Peter Meer, Synergsm n Low Level Vson, 16 th Internatonal Conference on Pattern Recognton (ICPR'0) [4] Rorn Comancu, Peter Meer, Mean Shft: A Rubust Approach Toward Feature Space Analyss, IEEE Transactons on Pattern Analyss and Machne Intellgence, Vol 4, No. 5, May 00. [5] Doug DeCarlo, Anthony Santella. Stylzaton and Abstracton of Photographs. In SIGGRAPH 00 [6] J. Frexenet, X. Munoz, D. Raba, J. Mart, and X. Cuf. Yet another survey on mage segmentaton. In 7 th European Conference on Computer Vson, volume III, pp , Copenhagen, Denmark, May 00 [7] James D. Foley, Andres van Dam, Steven K. Fener, John F. Hughes, Computer Graphcs: Prncples and Practce n C (nd Edton), Addson Wesley, [8] Th. Gevers and H.M.SG. Stokman, Robust photometrc nvarant regon detecton n multspectral mages. Internatonal Journal of Computer Vson 53(), pp , 003 [9] Bruce Gooch, Erk Renhard, Amy Gooch, Human Facal Illustratons: Creaton and Psychophyscal Evaluaton, ACM Transactons on Graphcs, Vol. 3, No. 1, January 004, Pages 7 44 [10] H. Levkowtz and G.T. Herman. Glhs: A generalzed lghtness, hue and saturaton color model. Computer Vson, Graphcs, and Image Processng: Graphcal Models and Image Processng, 55(4), [11] P. Meer, B. Georgescu: Edge detecton wth embedded confdence. IEEE Trans. Pattern Analyss and Machne Intellgence, 3, , 001 [1] S. E. Palmer. Vson Scence: Photons to Phenomenology. The MIT Press, Cambrdge, Mass. [13] E. Trucco,, and A. Verr, Introductory Technques for 3-D Computer Vson. Prentce-Hall [14] Jue Wang, Yngqng Xu, Heung-Yeung Shum and Mchael Cohen. Vdeo Toonng. SIGGAPH004 [15] Janlong Zhou and Klaus D. Tönnes. Focal regon-based volume renderng. In Proceedngs of the 9th Internatonal Workshop on Systems, Sgnals and Image Processng, pages , Manchester, U.K., November 00 a b c d Fgure 7. The statue of lberty under guded segmentaton. The source mage s and the segmentaton result shown n b takes brghtness as the perceptual gudance and the segmentaton result c takes hue value as the photometrc nvarant perceptual gudance, d s the combnaton of c and the bnary mage of the brghtness measure obtaned wth a user specfed threshold. 6

7 Fgure 8. More examples, left column are the orgnal mages, md column are the mages segmented by color spaces only, the rght column are segmentaton results wth perceptual gudance. 7