3D Rigid Facial Motion Estimation from Disparity Maps

Transcription

1 3D Rgd Facal Moton Estmaton from Dsparty Maps N. Pérez de la Blanca 1, J.M. Fuertes 2, and M. Lucena 2 1 Department of Computer Scence and Artfcal Intellgence ETSII. Unversty of Granada, 1871 Granada, Span ncolas@ugr.es Departmento de Informátca. Escuela Poltécnca Superor. Unversdad de Jaén Avenda de Madrd 35, 2371 Jaén.Span {jmf, mlucena}@ujaen.es 2 Abstract. Ths paper proposes an approach to estmate 3D rgd facal motons through a stereo mage sequence. The approach uses a dsparty space as the man space n order to represent all the 3D nformaton. A robust algorthm based on the RANSAC approach s used to estmate the rgd motons through the mage sequence. The dsparty map s shown to be a robust feature aganst local motons of the surface and s therefore a very good alternatve to the tradtonal use of the set of nterest ponts. 1 Introducton To date, many efforts have been made to study the problem of camera moton from features extracted from monocular or stereo mages [6,9,13]. The man approach estmates the moton by establshng correspondences between nterest ponts on each mage. There are two man shortcomngs of such an approach: frstly, t requres the set of nterest ponts on each mage to le on statc 3D surfaces of the scene; and secondly, the surfaces of the scene must be textured enough to allow nterest ponts to be estmated. When we approach the problem of estmatng 3D rgd facal motons from mages, we fnd that the problem of estmatng the rgd moton of a 3D surface wth many nstantaneous local deformatons s usually due to local facal motons [1,5,8]. Furthermore, t s well known that the surface of the face s not textured enough. Therefore, alternatves to the tradtonal use of the set of nterest ponts must be consdered. In ths paper, a homography between dsparty spaces s used to estmate 3D rgd motons. Dense dsparty maps are used as a feature from whch the homography parameters can be estmated. Snce we are nterested n studyng 3D object motons near the camera, we use the general perspectve camera model n order to analyze our mages. An mportant nstance of ths stuaton appears n 3D vdeoconferencng systems, where the 3D shape of the head and face of each partcpant must be refreshed n each nstant of tme, and the usual short dstance between cameras and surfaces ntroduces strong perspectve effects [12]. A. Sanfelu and J. Ruz-Shulcloper (Eds.): CIARP 23, LNCS 295, pp , 23. Sprnger-Verlag Berln Hedelberg 23

2 3D Rgd Facal Moton Estmaton from Dsparty Maps 55 In Secton 2, we ntroduce the geometrcal concepts of the dsparty space. In Secton 3, we study the rgd moton estmaton n the dsparty space. In Secton 4, dsparty map estmaton s dscussed. In Secton 5, experments carred out on mage data are shown. Fnally, n Secton 6, dscussons and conclusons are presented. 2 Stereo Images Let us consder a calbrated rectfed stereo rg,.e. the eppolar lnes are parallel to the x-axs. There s no loss of generalty snce t s possble to rectfy the mages of a stereo rg once the eppolar geometry s known [6]. We also assume that both cameras of the rectfed stereo rg have nternal parameters whch are smlar and known. Stereo reconstructon has been studed for years, and s now a standard topc n computer vson. Let us consder a rectfed mage par, and let (x,y) and (x,y ) be two correspondng ponts n that mage par. Snce the correspondng ponts must le on the eppolar lne, the relaton between the two ponts s x = x d (1) y = y where d s defned as the dsparty of the pont (x,y). From rectfed stereo mages, we can defne representaton spaces based on the projected coordnates that are equvalent to a 3D reconstructon of the ponts up to a homography of the 3D space [4]. These spaces are known as dsparty spaces. The equatons relatng the 3D coordnates (X,Y,Z) wth the dsparty coordnates n the case of orented and rectfed cameras are [13]: X x B x x y y x x Y = y x =, y =, x = (2) x x α α α Z 1 where x, y, x are the prncpal pont coordnates of the left and rght mage, respectvely, α and α are the focal dstance of the left and rght cameras, respectvely and B s the baselne of the stereo rg. All mage coordnates are expressed n terms of pxels. In ths paper, we use the dsparty space defned by the trple (x,y,d). From expresson (2), takng α=α, the homographc relatonshp between the 3D coordnates of a pont X=(X,Y,Z) T and ts assocated dsparty vector ( x, y, d) T can be expressed as x α y Z = d 1 α 1 X Y α B Z 1 (3) or n a shorter way as

3 56 N. Pérez de la Blanca, J.M. Fuertes, and M. Lucena 1 B X 1, W=( ) T x, y, d From equaton (3), t s clear that n the case of non-calbrated cameras each par of rectfed stereo mages provdes us wth the reconstructon of the surface beng maged up to projectvty. From the ntrnsc parameters of the stereo rg, the projectve reconstructon can be upgraded to metrc. (4) 3 Rgd Motons n the Dsparty Space Let us apply a rgd moton on the 3D data. If X and X represent the 3D coordnates of a pont before and after the moton, then X From expressons (4) and (5) we obtan R T X (5) = T τ λ = H 1 B R T T H 1 1 B τ = 1 τ 1 Equaton (6) descrbes the 3D homography * relatng the dsparty homogeneous coordnates of a pont before and after the moton. (6) 3.1 Nose on the Data An mportant feature of the dsparty space s that the nose assocated to the data vectors ( x, y, d) T under some assumptons can be consdered sotropc and homogeneous. The x, y dsparty coordnates are affected by the nose produced by the dscretzaton effect and wthout addtonal nformaton can be assumed equal for all pxels. The nose on d s assocated to the change n the gray level of the pxels n the stereo matchng process and could be estmated from ths process. We can therefore assume, that the noses assocated to x, y and d are ndependent. If we assume that the varance of d s of the same magntude as the varance of the dscretzaton error, the covarance matrx of the nose on each pont of our dsparty space s :=σ 2 I 3x3. In our case, apart from the above measurement errors, we also assume that n our scene there are ponts n moton. All the correspondences assocated wth these movng ponts are therefore potentally erroneous. In order to select pont correspondences whch are unaffected by the movng ponts, we use the RANSAC algorthm to select the subset of pont correspondences that are free of ths contamnaton.

4 3D Rgd Facal Moton Estmaton from Dsparty Maps Rgd Moton Estmaton Let (W W ) be a set of pont correspondences. The problem of estmatng the rgd moton parameters (R,T) from the set of ponts (W W ) amounts to mnmzng the error E 2 = Γ Γ Γ Γ Γ Γ Γ where = ( τ τ τ τ τ τ ) τ s the estmated Eucldean coordnate 4 vector for τ from (6), and : s the covarance matrx of the dsparty vectors. Here d 2 Γ T 1 Γ ( τ, τ ), d ( τ, τ ) = ( τ τ ) ( τ τ ) 2 (7) we assume an..d nose model. Equaton (6) shows that ths error functon s not lnear n the parameters for (R,T), so a non-lnear method has been used to estmate the vector of sx unknowns by parameterzng the rgd moton. Here we are nterested n the case of small rotatons (< 5 degree), so the rotaton matrx can be expressed as R=I+[Z] [, where I s the dentty matrx and [Z] [ represents the skewsymmetrc matrx assocated to the vector Z. In order to estmate the soluton vector (Z,T) T a quas-lnear teratve algorthm has been used on the normalzed mage coordnated [3]. An ntal soluton for the vector (Z,T) T can be calculated from equaton (6), solvng the lnear system that appears by consderng the equatons assocated to Eucldean coordnates of all the ponts τ and τ and assumng all λ=1. In the next teraton we recalculate the value of λ from the above soluton and agan solve equaton (6) for a new soluton. We terate untl convergence of the vector (Z,T) T. In our experence, three or four teratons are enough. Nevertheless, the presence of outlers n the correspondences between the dsparty maps degrades the estmaton consderably. In order to crcumvent ths problem a RANSAC based algorthm s proposed n Table 1. Ths algorthm makes a robust teratve lnear estmaton as a frst approach, but because of the nose n the dsparty estmaton, a non-lnear optmzaton step from the pxel color values s necessary. 4 Dsparty Map Estmaton In ths paper two dfferent dense dsparty maps are used. Frstly, we estmate the dsparty map for each stereo mage, and from ths we estmate a regon of nterest by applyng a bnary thresholdng operator on t. Secondly, we estmate the dense moton vector map assocated to every two consecutve left and rght mages, respectvely. In ths case we assume that the regon of nterest s the regon of movng pxels nearest the camera.

5 58 N. Pérez de la Blanca, J.M. Fuertes, and M. Lucena Table 1. Iteratve robust algorthmi. To estmate and normalze the set of dsparty vectors II. Repeat N teratons To choose n>=2 dsparty vectors randomly. For each vector calculate O, A and b. Solve OAX=b for X. Count the number of nlers.iii. To take the soluton wth hgher number of nlers as the best lnear soluton. IV. To mnmze the pxel color dfferences between mages by applyng the Levenberg-Mardquart algorthm from the lnear soluton. Fgure 1 shows how we estmate our regon of nterest on each stereo mage. In short, we segment the subset of movng ponts of the scene to a dstance of the camera, whch s less than a fxed threshold. In our case, the planar moton s calculated n pxel unts. In order to remove solated small regons we apply a sze flter. All the pctures shown correspond to the left mage of the stereo par. (a) (b) (c) (d) Fg. 1. Ths example corresponds to rotaton left-rght of the head. Pcture (a) represents the estmated stereo dsparty map, pcture (b) represents the x-moton dense map, pcture (c) represents the y-moton dense map, and pcture (d) represents the result of the unon of pcture (b) and pcture (c) ntersecton wth pcture (a). Dense dsparty maps from two mages s a very actve feld of research [1]. Very recently, new energy mnmzaton algorthms based on cut graphs was proposed [2][7]. These algorthms acheved a very good compromse between temporal effcency and accuracy of the estmaton [7]. Snce the mplementaton of these algorthms only depends on a free parameter, λ>, assocated to the scale of the estmaton [1], very dfferent estmatons can be acheved by varyng the λ value. Low values of λ provde us wth more accurate estmatons but a larger number of ponts wll be undefned. A scale combnaton scheme therefore provdes us wth a better estmaton. In our case, four dfferent scales (λ=3,5,1,3) have been consdered n order to estmate the dsparty maps. The combnaton scheme defned the dsparty value on each pxel as the value of the lowest scale n whch the dsparty s defned. For moton estmaton only the lowest scale has been used, snce the other

6 3D Rgd Facal Moton Estmaton from Dsparty Maps 59 scales do not contrbute much nformaton. In order to obtan as accurate a segmentaton as possble, there has been some loss n computatonal effcency. Fg. 2. The frst four columns of each row show the stereo dsparty map from a stereo mage, and the x-moton estmaton from two consecutve stereo mages, respectvely, for dfferent λ values. The last column shows the resultng estmaton from combnng the dfferent scales. All these mages correspond to the left mage of the stereo par. The frst row of Fgure 2 shows the stereo dsparty map estmaton from a stereo mage for dfferent values of λ joned to the fnal estmaton obtaned by combnng the dfferent scales. The second row shows the x-moton map estmaton from two consecutve stereo mages. It s possble to apprecate how the use of multple scales does not greatly mprove the frst scale estmaton n the case of moton estmaton. However, the combnaton of dfferent scales proves to be very useful when the stereo dsparty map s estmated. 5 Expermental Results Experments to estmate 3D rgd facal moton have been carred out from dfferent stereo mage sequences captured by a Pontgrey stereo camera (Bumblebee) watchng an actor movng hs face freely. A fxed wndow nsde the captured mages fxed the sub-mages of nterest. Our algorthm was appled to the mage sequence defned by the sub-mages. The proposed algorthm was appled on every two consecutve stereo mages n the sequence. In order to assess the goodness of the estmaton process we syntheszed a new sequence of mages by nterpolatng from the estmated motons and the orgnal sequence. Fgure 3 shows sx sampled mages to a dstance of ten samples, each, of a stereo sequence of our examples. It can be seen how the strength and unpredctablty of local facal motons makes t dffcult to use nterest ponts n the estmaton process. Fgure 4 shows how accurate the estmated moton for a partcular sequence s. We

7 6 N. Pérez de la Blanca, J.M. Fuertes, and M. Lucena compare the norm of the dfference between two consecutve mages, wth the norm of the resdual calculated by the dfference between an orgnal mage and ts correspondng synthetc. The large decrease of the norm of the dfference mage from the frst case to the second case, shows that the estmated moton s rght and precse enough. We should pont out that t s dffcult to vsualze the accuracy of the parameter estmaton from ths type of graph, but we prefer ths type because t s much more dffcult to apprecate small resdual motons by comparng eye statc pctures. Fg. 3. These pctures show local motons present n a standard stereo sequence. Fg. 4. Ths fgure shows four graphs each of whch s the norm of the gray level dfference pxel-by-pxel from two mages. The graphs Int.l-error and Int.r-error represent the case n whch the mages are two orgnal consecutve rght mages and left mages, respectvely, of the sequence. The graphs Ftt_l.error and Ftt_r.error represent the case n whch the two mages are the orgnal and syntheszed one, usng the proposed algorthm, for the rght and left mages, respectvely.

8 3D Rgd Facal Moton Estmaton from Dsparty Maps 61 6 Dscusson and Conclusons In ths paper a new approach to estmate the 3D rgd moton of a deformable surface s proposed. The algorthm we propose s accurate and fast enough snce no more than 3-4 lnear teratons plus 2 non-lnear teratons are needed for convergence. The use of stereo mages allows us to estmate the moton wthout the need for external nformaton. Ths result wll allow us to use ths approach to remove the rgd moton component from the dsparty vector to estmate local deformatons. Of course, n ths latter case and for large mage sequences, the accumulated error mght get very large. In order to avod ths stuaton, the present accuracy of the estmated moton based on two mages must be mproved. An alternatve n order to mprove estmaton would be the jont use of all mages n a bundle algorthm, but ths approach s napplcable n tme effcency demandng applcatons Acknowledgments. Ths work, has been fnanced by Grant IT from the Spansh Mnstry of Scence and Technology. References 1. Bascle, B., and Blake, A.: Separablty of pose and expresson n facal trackng and anmaton. In Proc. Int. Conf. Computer Vson Boykov,Y.,Veksler,O.,and Zabh,R.,: Fast approxmate energy mnmzaton va grpah cuts, IEEE Trans. PAMI vol-23,11, , Demrdjan, D., and Darell, T.,: Moton estmaton from dsparty mages, In Proc. ICCV1, Vancouver Canada,,21, vol-ii, Devernay, F. and Faugeras, O.: From projectve to Eucldean reconstructon. In Proceedngs Computer Vson and Pattern Recognton, , Fua, P.: Regularzed bundle-adjustment to models heads from mage sequences wthout calbraton data, Internatonal Journal of Computer Vson, 38(2), Hartley, R. and Zsserman, A.,: Multple Vew geometry n computer vson. CUP, Kolmogorov,V., and Zabh, R.,: Vsual correspondences wth occlusons usng graph cuts, In ECCV 2, Lecture Notes n Computer Scence 2352, 82 96,22 8. Lants, A., Taylor, C.J., Cootes, T.F. and Ahmed, T.: Automatc nterpretaton of human faces and hand gestures usng flexble models. In Internatonal Workshop on Automatc Face-and-Gesture Recognton, Pollefeys, M., Van Gool, L., Zsserman, A., and Ftzgbbon, A: 3D Structure from mages SMILE 2, Lecture Notes n Computer Scence 218, Sprnger, Scharsten,D., and Szelsk,R.,: A Taxonomy and evaluaton of dense two-frame stereo correspondence algorthms, IJCV, 47(1):7 42, Tarel, J.P.: Global 3D Planar Reconstructon wth Uncalbrated Cameras, A Rectfed Stereo Geometry, Machne Graphcs & Vson Journal, vol-6, 4, 1997, Valente.S. and Dugelay, J.L.: A vsual analyss/synthess feedback loop for accurate face trackng, Sgnal Processng Image Communcatons, 16, 21, Zhang,Z., Faugeras,O.: 3D Dynamc Scene Analyss,: A stereo based approach. Sprnger seres n Informaton Scence, 27, Sprnger-Verlag, 1992.