High Resolution Passive Facial Performance Capture

Transcription

1 High Resoluion Passive Facial Performance Capure Derek Bradley1 Wolfgang Heidrich1 Tiberiu Popa1,2 Alla Sheffer1 1) Universiy of Briish Columbia 2) ETH Zu rich Figure 1: High resoluion passive facial performance capure. From lef o righ: Acquisiion seup; one reference frame; he reconsruced geomery (1 million polygons); final exured resul; and wo differen frames. Absrac We inroduce a purely passive facial capure approach ha uses only an array of video cameras, bu requires no emplae facial geomery, no special makeup or markers, and no acive lighing. We obain iniial geomery using muli-view sereo, and hen use a novel approach for auomaically racking exure deail across he frames. As a resul, we obain a high-resoluion sequence of compaibly riangulaed and parameerized meshes. The resuling sequence can be rendered wih dynamically capured exures, while also consisenly applying exure changes such as virual makeup. CR Caegories: I.3.3 [COMPUTER GRAPHICS]: Picure/Image Generaion Digiizing and scanning; I.3.5 [COMPUTER GRAPHICS]: Compuaional Geomery and Objec Modeling Geomeric algorihms, languages, and sysems. Keywords: face reconsrucion, performance capure, markerless moion capure 1 Inroducion Facial performance capure is evolving ino a major ool for creaing realisic animaions in boh movie and game indusries. A pracical and versaile facial performance capure sysem should fulfill a number of requiremens. Firs, i should generae sequences of deailed meshes wih dynamic exure in order o capure boh geomeric deformaions of he face, as well as he corresponding changes in skin appearance due o sweaing or changes in blood circulaion. Second, he riangulaion of he meshes and heir mapping beween frames should be compaible over ime, such ha boh he geomery and he exure can be edied by an aris (e.g. by applying virual makeup), and hese changes can be propagaed over ime. Finally, he capure process iself should be simple and auomaic, and should no require separae geomery scans or excessively expensive hardware. Reconsrucing a human face is challenging, since mos faces do no have sufficien medium-scale exure o esablish dense correspondences beween differen viewpoins. For his reason, commonlyused mehods ypically involve eiher srucured lighing [Zhang e al. 2004; Wang e al. 2004], special makeup [Furukawa and Ponce 2009], or markers [Bickel e al. 2007; Ma e al. 2008] o rack he geomery. This makes i difficul o simulaneously capure boh geomery and exure, hus requiring eiher inpaining of markers or sacrificing emporal resoluion by saggering srucured ligh wih uniform ligh. Addiionally, hese acive mehods can be uncomforable for he acors, affecing heir performance. Many of hese echniques also require an iniial laser scan of he face, while he ohers suffer from low-resoluion reconsrucions. In his paper, we presen a fully passive mehod for facial performance capure ha saisfies he crieria oulined above. By using only a camera array and uniform illuminaion, our seup is less inrusive o he acors. We require no markers, no face pain, no fluorescen makeup, no srucured ligh, and no laser-scanned model, ye are sill able o reconsruc high resoluion ime-varying meshes a 30 frames per second. Our passive seup also allows us o reconsruc high-resoluion, ime-varying exure maps ha can capure poenial changes in skin appearance due o, for example, blushing or sweaing of he acor. Our reconsrucion is made possible by a number of conribuions: 1. A novel high-resoluion acquisiion seup ha allows us o use pores, blemishes and hair follicles as rackable feaures. 2. A high-qualiy sereo reconsrucion algorihm, exending a curren sae-of-he-ar echnique [Bradley e al. 2008a] o include new dispariy consrains ha work wih our seup. 3. Mos significanly, an auomaic surface racking mehod based on opical flow. This mehod suppors auomaic drif correcion and edge-based mouh racking, which ogeher yield realisic, ime-varying, exured facial models. Our capure sysem consiss of he hree main componens shown in Figure 2: Acquisiion seup - Our seup consiss of 14 high definiion video cameras, arranged in seven binocular sereo pairs. Each pair is zoomed-in o capure a small pach of he face surface in high deail under brigh ambien illuminaion (Secion 3).

2 Acquisiion Muli view Reconsrucion Figure 2: Overview of our algorihm. Geomery & Texure Tracking Muli-view reconsrucion - We use an ieraive binocular sereo mehod o reconsruc each of he seven surface paches independenly, and hen combine hem ino a single highresoluion mesh. The zoomed-in cameras allow us o use skin pores, hair follicles and blemishes as surface exure o guide he sereo algorihm, producing meshes wih roughly 1 million polygons (Secion 4). Geomery and exure racking - In order o consisenly rack geomery and exure over ime, we choose a single reference mesh from he sequence, and compue a mapping beween i and every oher frame by sequenially using opical flow. The observed pores and oher surface deails no only serve o provide accurae per-frame reconsrucions, bu also allow us o compue cross-frame flow. Drif caused by ineviable opical flow error is deeced in he per-frame exure maps and correced in he geomery. In order o accoun for he high-speed moion generaed by alking, he mapping is guided by an edge-based mouh-racking process (Secion 5). 2 Relaed Work Previous echniques for facial performance capure can largely be divided ino he following caegories: mehods ha use markers or special face pain o guide he reconsrucion, srucured ligh approaches, and finally mehods ha fi parameric face models o observed video sequences. Markers and Face Pain. Tradiional marker-based face capure daes back o Williams [1990], and consiss of covering he face wih a large number of black dos or fluorescen colors [Guener e al. 1998; Lin and Ouhyoung 2005] in order o rack he geomery. Bickel e al. [2007] augmen he radiional mehods o incorporae muliple scales of geomery and moion using markers, face pain, and an iniial scan. Mos recenly, Furukawa and Ponce [2009] presen a face capure echnique ha deforms a laser-scanned model o mach a highly-pained face while regularizing non-rigid angenial deformaions. Unforunaely, hese echniques have rouble reconsrucing accurae per-frame face exures, and require expensive hardware o perform iniial scans. Srucured Ligh. Anoher approach o facial performance capure is o combine cameras wih a leas one projecor ha cass a srucured ligh paern ono he face in order o provide dense surface exure. Zhang e al. [2004] use space-ime sereo wih srucured ligh o reconsruc emporally smooh deph maps of a face. They hen fi a deformable emplae model a each ime sep using opical flow. Wang e al. [2004] projec phase-shifed color-fringe paerns ono he face and acquire he 3D shape in real-ime. They esablish correspondences beween frames using a muli-resoluion model fiing approach. However, he resuling meshes have insufficien resoluion (a mos 16K verices) for capuring fine-scale facial deails such as wrinkles. Ma e al. [2008] achieve high-resoluion reconsrucions by inerleaving srucured ligh wih spherical gradien phoomeric sereo [Ma e al. 2007] using he USC Ligh Sage. New facial performances are hen synhesized using a marker-based, daa-driven approach. However, his mehod requires expensive hardware. Srucured ligh approaches are unaracive because hey can be disracing o he acor, and hey suffer from he inabiliy o reconsruc face exures wihou sacrificing emporal resoluion by inerleaving ambien illuminaion wih he srucured ligh. Parameric Models from Video. The goal of his caegory of mehods is o deermine he parameers of a deformable face model from observing a video sequence wihou markers or srucured ligh [Li e al. 1993; Essa e al. 1996; DeCarlo and Meaxas 1996; Pighin e al. 1999]. However, he resuling face reconsrucions end o be very low resoluion, lacking any person-specific deails. In a relaed work, Blanz e al. [2003] re-animae faces in video by parameerizing a daabase of differen laser-scanned faces and expressions. They can hen esimae he 3D shape and pose of new faces in video images wih a parameer fiing algorihm. Bu like he oher parameric approaches, he final models end o lack facial deails. Commercial Sysems and Oher Projecs. A number of commercial sysems have been developed for facial performance capure. Vicon s marker-based sysem 1 and Mova s fluorescen makeup CONTOUR Realiy Capure 2 are wo prominen examples, while Dimension Imaging 3D is among he firs o use markerless facial reconsrucion in indusry 3. Recenly, Alexander e al. [2009] creaed a phooreal facial modeling and animaion sysem in he Digial Emily Projec. Finally, Borshukov e al. [2003] recreae acors for The Marix Reloaded using heir Universal Capure sysem. These sysems boh sar wih laser-scanned models. The approach of Borshukov is mos similar o ours. Opical flow and camera riangulaion advances a face model over ime, and a ime-varying exure map is compued from muliple videos. Unlike our auomaic approach, however, opical flow errors and drif are correced using edious manual geomery reshaping. To our knowledge, ours is he firs face capure mehod wih fullyauomaic emporal reconsrucion, rivaling curren sae-of-he-ar echniques, bu wihou he need for markers, face pain, expensive hardware or srucured ligh. I is worh noing ha concurren o our work, Beeler e al. [2010] presen a similar echnique for passive face reconsrucion ha capures pore-scale geomery of saic faces. Our auomaic surface racking mehod for generaing emporally compaible animaions could complemen heir approach. 1 Vicon MX CONTOUR Realiy Capure

3 Figure 3: Ieraively consrained muli-view sereo removes ouliers by ieraively ighening deph consrains. 3 Acquisiion Seup Our acquisiion seup consiss of 14 high definiion Sony HDR-SR7 cameras arranged in an array in fron of he acor (see Figure 4). The cameras are geomerically calibraed [Bradley and Heidrich 2010] and opically synchronized [Bradley e al. 2009]. We use nine LED ligh fixures, each wih 192 LEDs, o provide boh brigh, uniform illuminaion, as well as he camera synchronizaion. Figure 4: Schemaic view and phoo of our acquisiion seup. As we menioned previously, reconsrucing a human face is challenging because mos faces do no have sufficien medium-scale surface exure. This is he main reason why previous work relies on hand-placed markers or srucured ligh paerns. However, his assumpion does no hold if we increase he resoluion of he acquisiion sysem and capure images of very small face deails such as pores, freckles and wrinkles. To his end, we arrange he cameras in seven sereo pairs and maximize he opical zoom level in order o observe fine-scale surface deails, providing a naural surface exure for reconsrucion. Figure 4 shows an illusraion of he seup and he seven face regions observed by he sereo pairs, along wih a phoo of he acual seup. Noe ha here is significan overlap beween he face regions, even hough i is no shown in he illusraion. Figure 5 shows a video frame capured from one camera, demonsraing he high resoluion surface deails. We also have an addiional reference camera ha is no zoomed-in and no used for processing. This camera provides a normal video for comparison wih our resuls. Figure 5: One HD video frame showing ha pores can be used as naural surface exure for face reconsrucion. Preparing an acor for capure requires very lile work, since we do no place markers on he face or require an iniial scan. We do use off-he-shelf foundaion makeup o reduce speculariy in he case of oily skin, however his is common pracice for preparing acors for any performance. 4 Muli-View Reconsrucion To reconsruc a mesh for each face frame, we perform binocular sereo for each of he seven facial regions, resuling in seven overlapping deph images. The naural surface exure obained from our capure seup provides sufficien deail for sereo reconsrucion. The seven deph images are hen merged ino a single riangle mesh using he muli-view reconsrucion sysem of Bradley e al. [2008a]. This sysem was successfully used for garmen capure [Bradley e al. 2008b], which has similar requiremens on accuracy and efficiency for capuring deformable geomery. However he mehod of Bradley e al. [2008a] is designed for 360 reconsrucion, where he visual hull of he objec can be precompued and used o reduce ouliers in he binocular reconsrucions. Our camera seup prevens us from applying he echnique exacly as described, because we observe only paches of he face and do no have full 360 coverage. Thus we canno compue a visual hull. Wihou he visual hull consrain, he resuling deph images can conain many ouliers (Figure 3, c). We resolve his problem by adoping an ieraively consrained binocular reconsrucion approach, designed o ieraively remove ouliers in he reconsrucion. We apply his mehod o each of he seven camera pairs. In he firs pass of binocular reconsrucion, we enforce a loose deph consrain ha corresponds o he maximum reconsrucion volume for he face pach (measured by hand). The resuling deph image has many valid samples bu also conains ouliers. Figure 3 (a) and (b) shows an example sereo pair, and he deph ouliers can be seen in he corresponding poin cloud in Figure 3 (c). In order o reduce ouliers, we ighen he iniial deph consrains for he nex ieraion. To his end, we perform Gaussian smoohing on he curren deph image using a large kernel size, creaing an over-smooh approximaion of he surface wih fewer ouliers. This smooh deph image is used o compue per-pixel consrains for a second pass of binocular sereo. During he second pass, we process only he pixels whose deph in he firs reconsrucion violaes he new consrains. If he deph is wihin an accepable disance from he smooh surface hen we do no re-process he pixel. By ighening he deph consrains, he deph image compued in he second pass has fewer ouliers (Figure 3, d). We repea he smoohing process on he new deph image o esablish new consrains for he nex pass, ieraing beween sereo maching and consrain ighening unil convergence. In pracice, we found ha hree ieraions was sufficien for all reconsrucions (Figure 3, e). Ieraively Consrained Binocular Reconsrucion. In his approach, we assume ha he rue surface lies wihin a small disance of he over-smoohed one in each ieraion. This holds rue if he surface does no conain very high curvaure or sharp feaures, which is he case for faces. We also assume ha ouliers have small local suppor, so ha hey are removed by he smoohing sep. Afer applying ieraive binocular reconsrucion for each camera pair, we obain seven corresponding poin cloud reconsrucions. The poin clouds are hen merged ino a single dense poin cloud. On average, we reconsruc approximaely 8-10 million poins per face. These poins are downsampled, filered and Pair Merging.

4 5.1 Reference Mesh We sar each performance wih a neural facial expression, from which we creae our reference mesh. The firs reconsrucion, G0, (shown in Figure 6 lef) is manually edied o remove any ouliers caused by hair, and hen a 2D parameerizaion of he geomery is compued. We use LSCM [Le vy e al. 2002] o generae he parameerizaion because i successfully deals wih he holes in our iniial reconsrucion. The holes are filled in 2D by creaing small Delaunay riangulaions [Shewchuk 1996], and he new 3D geomery is creaed as a membrane surface, C 1 coninuous wih he surrounding geomery, using he Laplacian formulaion of Bradley e al. [2008b]. Finally, a sli is cu in he mesh for he mouh. The resul is he firs mesh M 0 of he final sequence. The 2D parameerizaion compued here will become he domain for he exure map. Noe ha his parameerizaion remains consan for he enire sequence. Tha is o say, as a verex vi of he mesh moves over ime, is exure coordinaes never change. Figure 6: Muli-view reconsrucions for hree differen frames. riangulaed using he sysem of Bradley e al. [2008a] wihou furher modificaion. Our final meshes have approximaely 500K verices (1 million riangles). Each frame of a capure sequence can be reconsruced independenly, so we perform he reconsrucions in parallel. In he following we refer o hese iniial meshes as G. The reconsrucion of a selecion of frames is shown in Figure 6. 5 Geomery and Texure Tracking Given he per-frame reconsrucions, he nex sep is o reconsruc he moion of he face by racking he geomery and exure over ime. We explicily compue a sequence of compaible meshes wihou holes, which allows ariss o laer edi boh he geomery and he exure, and o propagae hese modificaions consisenly over ime. Given he iniial per-frame reconsrucions G, we would like o generae a se of compaible meshes M ha have he same conneciviy as well as explici verex correspondence. Tha is o say, we desire one mesh ha deforms over ime. In order o creae high-qualiy renderings, we also require per-frame exure maps T ha capure appearance changes such as wrinkles and sweaing of he acor. We propose an opical flow based approach for moion reconsrucion, as illusraed in Figure 7. The basic mehod works as follows: saring wih a single reference mesh M 0, generaed by manually cleaning up he firs frame G0 (Secion 5.1), we compue dense opical flow on he video images and use i in combinaion wih he iniial geomeric reconsrucions G o auomaically propagae M 0 hrough ime (Secion 5.2). A each ime sep, we compue a highqualiy 2D face exure T from he video images (Secion 5.3). In heory his basic mehod can reconsruc he face moion and produce a emporally consisen animaion. However, he pracical limiaions of opical flow can pose problems. For insance, opical flow is prone o errors during rapid deformaions such as lip movemen during speech, and can be inaccurae for a variey of oher reasons including occlusions, insufficien image deails, and appearance changes such as he formaion of wrinkles. Furhermore, since he basic mehod processes he frames sequenially, even he smalles error will accumulae over ime, causing he geomery o drif. Temporal drif is unaccepable, as i will desroy he consisen mapping beween frames ha we aim o reconsruc. We overcome hese issues wih wo improvemens o he basic mehod, which aim o sabilize he animaion hrough explici mouh racking and exure-based drif correcion (Secion 5.4). As a final sep, we perform smoohing o remove unwaned noise in he animaion (Secion 5.5). 5.2 Frame Propagaion (Basic Mehod) We compue opical flow [Bougue 1999] over he whole sequence for each of he 14 video cameras individually. Using his flow and he iniial reconsrucions G, we can now propagae M 0 forward in ime o produce our oupu sequence. The process is illusraed in Figure 8, and i proceeds as follows. For each verex vi 1 of M 1 we projec he verex ono each camera c in which i is visible (i.e. inside he field of view of and no occluded). Le pi,c be his projeced pixel. We hen look up he 2D flow vecor ha corresponds o pi,c and add he flow o ge a new pixel locaion p0i,c. Back-projecing from p0i,c ono G gives us a guess for he new ver ex locaion, which we call v i,c. The illusraion in Figure 8 has exaggeraed iner-frame moion for beer visualizaion. vi 1 v i,c M 1 pi,c G p0i,c pi,c c p0i,c c Figure 8: Compuing verex posiions for he nex frame, using percamera opical flow (wih exaggeraed moion for visualizaion). We require a leas wo cameras o agree on he new verex locaion. We say ha cameras c1 and c2 agree if k v i,c v i,c k< 1mm. 1 2 (1) v i The compued verex locaion is hen a weighed average of he n per-camera guesses ha agree: v i = n X wi,c v i,c, (2) c=1 where wi,c is he do produc beween he surface normal a v i,c and he vecor from here o c. The difference beween he new verex locaion and is previous locaion can be considered a 3D flow

5 (Sec. 4) (Sec. 4) Opical Flow (Sec. 4) Opical Flow Opical Flow G1 G2 G3 Reference Frame (5.1) M0 M1 Frame Propagaion (5.2) T0 M2 Frame Propagaion (5.2) Frame Propagaion (5.2) T1 2D Texure (5.3) M3 T2 T3 Figure 7: Overview of our geomery racking algorihm. vecor, which we denoe δi. If no enough cameras agree on a new verex locaion, for example if a verex falls on a hole in he nex frame, hen he 3D flow from neighboring verices is inerpolaed. We assume ha he moion of he face is spaially smooh, so neighboring verices have similar 3D flow. The inerpolaion is achieved by solving a simple leas-squares Laplacian sysem on he surface for all verices ha were no updaed (all updaed verices remain fixed): min k δi k2. (3) Finally, we apply regularizaion o he mesh in order o avoid possible riangle-flips and remove any unwaned arifacs ha may have been presen in he iniial reconsrucion. Following he regularizaion mehod of de Aguiar e al. [2008], we again solve a leassquares Laplacian sysem using coangen weighs and he curren posiional consrains v i. Thus, we generae he final mesh M by minimizing arg min{k vi v i k2 +α k Lv Lv 0 k2 }, v (4) To compue he exure for frame, we sar by projecing each riangle of M ono he camera ha observes i bes, as deermined by he do produc beween he riangle normal and he camera direcion. The camera pixels corresponding o he projecion are hen copied o he corresponding 2D riangle in he exure domain. Figure 9 shows he compuaion of a face exure. We visualize he conribuion from each camera as a grayscale pach image in Figure 9 (middle-lef). Figure 9 (middle-righ) shows he iniial exure resul afer copying he pixels from he camera images. Since he cameras are no radiomerically calibraed, differen exure paches can have drasically differen skin ones. To compue he final exure, shown in Figure 9 (far righ), we apply Poisson image ediing [Pe rez e al. 2003] similar o Mohammed e al. [2009]. We sar wih he larges pach and ieraively add adjacen paches unil he exure image is complee. For each new pach we compue x- and y-gradiens inside he pach and solve a Poisson equaion o find a new pach ha maches he gradiens as closely as possible, while also obeying he boundary condiions se by oher compleed paches. Finally, in order o have emporally consisen exures, we use he previous exure T 1 as per-pixel sof consrains when solving he Poisson equaion. where L is he coangen Laplacian marix. The parameer α conrols he amoun of regularizaion, and is se o 100 for all reconsrucions. 5.3 Compuing 2D Texure A each ime sep, in addiion o reconsrucing geomery, we also compue a high-resoluion 2D exure T for rendering. Since we do no use markers or face pain, our exure images are rich in deail, conaining per-frame appearance changes due o, for example, blushing or sweaing of he acor. As poined ou by Borshukov e al. [2003], hese exural variaions are very imporan for creaing believable facial renderings. All of our 14 HD cameras are used o compue a single exure ha covers he enire surface. This allows us o creae very highresoluion exures, in he order of 8-10 megapixels, for exreme zooms (see Figure 17). For oher purposes, lower resoluion exures may be sufficien (e.g as used in mos of our examples). The domain of he exure image is given by he 2D parameerizaion of he mesh (Secion 5.1). Every verex of he mesh has unique 2D coordinaes in he parameer domain, yielding a one-oone mapping beween 2D and 3D mesh riangles. Figure 9: 2D exure generaion. From lef o righ: reference image, camera conribuion image, iniial exure, final exure. Two addiional exures for he same sequence are shown in Figure 10. Noice how he exure domain remains fixed even hough he 3D face undergoes subsanial deformaions, including opening of he mouh. The only differences in he exures is changes in skin appearance caused by wrinkles, blushing or sweaing of he acor. 5.4 Tracking Enhancemens In general, he basic opical flow-based racking echnique described so far produces realisic animaions of face deformaion. For mos of he face, opical flow vecors are boh dense and accurae, since our capure seup provides naural high-resoluion surface feaures which easily guide he flow compuaion. However,

6 poins. We encode he barycenric coordinaes in a sparse marix B which has similar srucure o he Laplacian marix, excep ha i conains rows only for verices ha are adjacen o a consrain riangle. Le P 0 be he se of 3D consrain poins deermined from he back-projecion, hen P 0 = Bv 0. Figure 10: Texure resuls for wo differen frames, including reference images. opical flow can fail during very fas moion such as rapid mouh deformaions, and minor inaccuracies can accumulae over ime leading o emporal drif. We resolve hese problems auomaically by enhancing he basic echnique wih an explici mouh racking algorihm, along wih a mehod for deecing and correcing emporal drif Mouh Tracking To perform auomaic mouh racking, we inroduce posiional consrains for a sparse se of poins around he mouh a each ime sep. The posiional consrains are compued in image-space, and hus do no map direcly o mesh verices. Insead, hey are incorporaed as barycenric consrains on mesh riangles. The consrain poins are deermined by racking he mouh in a single camera (eiher one of he wo green cameras in Figure 4). We perform edge deecion [Canny 1986] on each frame wihin a user-specified region-of-ineres (ROI). The region should conain he mouh hroughou he sequence bu avoid oher edges caused by surrounding wrinkles or he silhouee of he face. If he sequence conains oo much global face moion o conain he mouh in a single ROI hen mouh racking can be performed in emporal segmens wih differen ROIs. For each frame we perform a simple analysis of he deeced edges o locae he mouh. Figure 11 shows a few differen frames of mouh racking. Deeced edges are shown in whie, and he ROI is indicaed by he blue recangle. If we consider he image as a se of rows and columns, we sar by choosing he wo corners of he mouh (shown as red poins) as he minimum and maximum columns ha conain an edge pixel. We hen deec he op and boom lips by uniformly sampling a sparse number of columns beween he mouh corners, and selecing he minimum and maximum rows a each column ha conain edge pixels (shown as green poins). Empirically we found ha 30 sample columns were sufficien. These 62 pixels hen become he mouh consrains for his frame. We process each frame in he same manner, yielding an explici emporal correspondence beween each of he individual consrains. Since edge deecion is nooriously unsable, we smooh he consrains boh spaially and emporally o remove ouliers. Alhough our mouh racking echnique is raher simple, we found he resuls o be quie robus, as we show in he boom row of Figure 11. Figure 11: Mouh racking hrough edge deecion. The green and red poins become consrains in geomery racking. We back-projec he consrained pixels ino he reference frame M 0 o deermine he se of consrain mesh riangles and he corresponding barycenric coordinaes for each of he mouh consrain (5) Throughou he sequence B remains fixed. The per-frame mouh consrains are used o compue 3D consrain poins P by backprojecing ono he iniial reconsrucions G. The mouh consrains guide he regularizaion from Secion 5.2, replacing Equaion 4 wih arg min{k vi v i k2 +α k Lv Lv 0 k2 +β k Bv P k2 }, v (6) where β conrols how much we consrain he mouh. We use a high weigh, β = 104, since he mouh can deform quie rapidly, causing large errors in he basic opical flow approach. Our mouh racking procedure alleviaes hese errors, and hus is an essenial par of our mehod for generaing realisic facial animaions Drif Correcion I is well-known ha opical-flow based racking mehods suffer from accumulaion of error, known as drif [DeCarlo and Meaxas 2000; Borshukov e al. 2003]. DeCarlo and Meaxas [1996] solve his problem by combining opical flow wih edge informaion, and Borshukov e al. [2003] rely on manual inervenion. A key feaure of our mehod is ha we are able o deec and correc drif in he 3D animaion auomaically, using he exure domain of he faces. Drif ypically occurs because opical flow is compued beween successive video frames only. If i were possible o accuraely compue flow beween he firs video image and every oher frame, here would be no accumulaion of error. Unforunaely, emporally disan video images in a capure sequence are usually oo dissimilar o consider his opion. In our case, however, he exure domain of he mesh remains consan over ime, which means ha he compued per-frame exure images are all very similar. Any emporal drif in he 3D geomery appears as a small 2D shif in he exure images, which can easily be deeced, again by opical flow. To incorporae drif correcion, we employ a simple modificaion o he basic racking mehod described in Secion 5.2. Afer compuing he geomery M and exure T for a given frame, we compue opical flow beween he exures T 0 and T. This flow (if any is deeced) is hen used o updae M on a per-verex basis using he direc mapping beween he geomery and he exure. Any shif in exure space becomes a 3D shif along he mesh surface. Afer updaing he verices o accoun for drif we apply regularizaion again (Equaion 6), o avoid possible riangle flips. The only problem ha remains is ha, if significan appearance changes such as wrinkles have occurred in he curren frame, opical flow beween T 0 and T can fail, resuling in large flow errors. However, since we expec drif o appear gradually, he flow beween T 0 and T should never be more han a few pixels. Larger flow vecors are discarded as ouliers. Sill, face regions ha conain hese appearance changes may incur drif, as wrinkles can be presen for a significan number of frames. In hese regions, we deec drif more locally by compuing he flow beween T k and T, and updaing he geomery accordingly. We choose k o be small, so ha boh frames ( k) and conain similar appearance, such as he same wrinkles, allowing flow o be compued accuraely. On he oher hand, k mus be large enough so ha drif can accumulae and be deeced. In all reconsrucions, we found ha k = 5 was an

7 appropriae rade-off. The decision o swich from global o local drif correcion is made auomaically, on a per pixel basis, when here is no valid opical flow beween T 0 and T. By performing local drif correcion we are, in effec, only slowing down he drif accumulaion raher han removing i. However, his approach does sabilize he animaion unil he wrinkles disappear, a which ime normal drif correcion is auomaically resumed. 5.5 eye regions, as shown in Figure 13. These eye regions are marked manually in he firs frame and are hen propagaed auomaically for he res of he sequence. Pos-Processing As a final sep, we pos-process he sequence o provide a smooh, realisic facial animaion. As wih any sereo reconsrucion mehod, spaial noise can appear in he resuling geomery due o, for example, sligh inaccuracies in camera calibraion (see Figure 12-boom lef). We wish o smooh he face meshes o remove his noise, bu avoid removing he spaial feaures and wrinkles ha define he face. To accomplish his, we inroduce saliencybased smoohing, a echnique for smoohing less-salien regions of he face while preserving more-salien feaures. Saliency is compued in he exure image T hrough a simple analysis of local hisograms. Kadir and Brady [2001] remark ha areas of an image wih high saliency end o have flaer disribuions in he local hisogram of inensiies. Following his principle, we mark a pixel in T as non-salien if is local hisogram conains a single srong peak. All oher pixels are salien. We compue hisograms in local 15x15 windows, quanized o 8 inensiy bins and use simple hresholding o deermine if a peak exiss. Figure 12 (op row) shows one of he exure images and he compued saliency mask, where salien pixels are whie. The saliency mask is hen used o consrain salien verex posiions in a Laplacian smoohing sep. The resul is shown in he boom row of Figure 12. Noice ha he eyebrows, mouh and cheek deformaion were no affeced by he smoohing. Saliency-Based Smoohing. Figure 13: Local smoohing o improve he eye regions. We end wih a single pass of Gaussian smoohing in he emporal domain o preven emporal flicker. In pracice we found ha emporal noise was minimal, so we use a small smoohing radius of only hree frames. Temporal Smoohing. Afer all pos-processing, 2D exures are re-compued since he face geomery has changed. 6 Resuls We now show resuls for a number of facial performances given by differen people. All of our resuls were rendered wih Renderman, using he MakeHuman skin shader in Pixie4. We encourage he reader o also view he accompanying video, which furher demonsraes our resuls. Figure 18 conains six frames from he firs sequence, rendered in a number of differen ways o highligh he resuls. The op row is he reference fooage. In he second row we show he geomery wihou a exure map. Here we can see he exac geomeric deails ha form each facial expression. The hird row is rendered using a saic checkerboard exure. This visualizaion shows how he skin sreches and compresses during deformaion, for example when raising he eyebrows in he second image from he lef. The sabiliy of he checker paern over ime also indicaes ha we have achieved a drif-free reconsrucion. The fourh row shows our high-qualiy rendering using he capured per-frame exures. Finally, he las row demonsraes how virual makeup can be applied o he sequence. Here, an aris would edi he makeup in he firs frame and he edis would be propagaed o he res of he sequence auomaically. Once a facial performance has been capured, we can also render i from arbirary viewpoins or lighing condiions, as we show in Figure 19. In he fifh column of Figure 18, noe ha he mouh of he acor is open, resuling in a hole where he eeh should be. Like mos mehods for facial performance capure, we do no reconsruc he inside of he mouh. Alhough his effec can be disracing, making he enire face appear differen from he reference image, we illusrae ha he face reconsrucion is sill accurae by manually composiing he eeh from he reference frame ino he resul, as shown in Figure 14. The final resul including he eeh now maches he reference frame and is very compelling, indicaing an area of fuure work o simulaneously reconsruc ime-varying eeh models wih he res of he face. Figure 12: Saliency-based spaial smoohing. Top: an inpu exure and he saliency map. Boom: before and afer smoohing. As we saw in he iniial reconsrucions (Figure 6) and again in Figure 12, he eye regions are no reconsruced well. This is because eyes are oo specular, and surrounding eye-lashes are hin hairs ha only add noise o he surface. We alleviae his problem by performing localized smoohing in he Relaxing he Eye Geomery. We show he versailiy of our approach by capuring wo oher acors, one male and one female. Resuls are shown in Figure 15 and Figure 16, including he pure geomery resul and he high-qualiy exured rendering. Two exreme zoom renderings are shown in Figure 17(middle and righ), using a 10 megapixel exure ha we reconsruced from he video images

8 Figure 14: Adding he eeh creaes a compelling resul, indicaing an area for fuure research. Figure 16: Capure resuls for ye anoher sequence, including he reference frames (lef), pure geomery resul (cener), and highqualiy rendering wih exure (righ). Figure 17: 10 megapixel exures allow exremely close zoom renderings (middle and righ). Figure 15: Capure resuls for anoher sequence, including he reference frames (lef), pure geomery resul (cener), and high-qualiy rendering wih exure (righ). 7 Conclusion In his paper we presen a purely passive mehod for facial performance capure. Unlike previous mehods ha require markers, face pain, srucured ligh, or expensive hardware, we use only a camera array and uniform illuminaion. Noneheless, we are able o reconsruc high resoluion, ime-varying meshes a 30 frames per second. The absence of markers and srucured ligh allow us o capure deailed, per-frame exures and creae high-qualiy renderings. One of he keys o our approach is our novel high-resoluion acquisiion seup. We are able o use naural skin blemishes, hair follicles and pores boh for esablishing deailed geomeric reconsrucions, and also for racking he face over ime. Our geomery and exure racking mehod is fully auomaic, and includes robus emporal drif correcion. While he small facial deails are visible in he video images, spaial geomery regularizaion and emporal smoohing preven us from reconsrucing he fines pore-scale geomery. These smoohing seps are required o overcome inaccuracies in he iniial reconsrucions and in he opical flow vecors. However, we do provide much higher resoluion han previous passive mehods [Li e al. 1993; Essa e al. 1996; DeCarlo and Meaxas 1996; Pighin e al. 1999]. Also noe ha fine wrinkles and pores can be added o he geomery in a pos-process similar o Beeler e al. [2010], and we consider his fuure work. Our mehod is versaile, which we demonsrae by reconsrucing hree differen facial performances given by differen acors, and including very differen deformaions. To our knowledge, his paper presens he firs auomaic echnique o reconsruc high-qualiy facial performances wihou he need for markers, face pain, or srucured ligh. The main limiaion of our echnique is ha very fas moion can lead o incorrec geomery racking due o moion blur and inaccurae opical flow. Furhermore, since our mehod processes frames sequenially, an error in racking could cause an early erminaion of he face sequence. Addiionally, our mehod is no designed o reconsruc facial hair, and we require manual processing of one frame of he sequence o build he reference mesh. In he fuure, we plan o explore mehods for auomaically compleing he face model by capuring he eeh. In addiion, correcly capuring he eye geomery, including eyelashes and eyebrows, would produce even more realisic resuls, paricularly for high-qualiy specular rendering of he eyes.

9 Acknowledgmens We would like o hank our acors: Ben Ceccheo, Ian Souh- Dickinson and Jennifer Fernquis. This work was funded in par by he Naural Sciences and Engineering Research Council of Canada and he GRAND Nework of Cenres of Excellence of Canada. References ALEXANDER, O., ROGERS, M., LAMBETH, W., CHIANG, M., AND DEBEVEC, P The digial emily projec: phooreal facial modeling and animaion. In ACM SIGGRAPH Courses, BEELER, T., BICKEL, B., SUMNER, R., BEARDSLEY, P., AND GROSS, M High-qualiy single-sho capure of facial geomery. ACM Trans. Graphics (Proc. SIGGRAPH). BICKEL, B., BOTSCH, M., ANGST, R., MATUSIK, W., OTADUY, M., PFISTER, H., AND GROSS, M Muli-scale capure of facial geomery and moion. ACM Trans. Graphics (Proc. SIGGRAPH), 33. BLANZ, V., BASSO, C., VETTER, T., AND POGGIO, T Reanimaing faces in images and video. Compuer Graphics Forum (Proc. Eurographics) 22, 3, BORSHUKOV, G., PIPONI, D., LARSEN, O., LEWIS, J. P., AND TEMPELAAR-LIETZ, C Universal capure: image-based facial animaion for he marix reloaded. In ACM SIGGRAPH Skeches & Applicaions. BOUGUET, J.-Y Pyramidal implemenaion of he lucas kanade feaure racker: Descripion of he algorihm. Tech. rep., Inel Corporaion, Microprocessor Research Labs. BRADLEY, D., AND HEIDRICH, W Binocular camera calibraion using recificaion error. In Canadian Conference on Compuer and Robo Vision. BRADLEY, D., BOUBEKEUR, T., AND HEIDRICH, W Accurae muli-view reconsrucion using robus binocular sereo and surface meshing. In Proc. CVPR. BRADLEY, D., POPA, T., SHEFFER, A., HEIDRICH, W., AND BOUBEKEUR, T Markerless garmen capure. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3, 99. BRADLEY, D., ATCHESON, B., IHRKE, I., AND HEIDRICH, W Synchronizaion and rolling shuer compensaion for consumer video camera arrays. In Inernaional Workshop on Projecor-Camera Sysems (PROCAMS 2009). CANNY, J A compuaional approach o edge deecion. IEEE Trans. Paern Anal. Mach. Inell. 8, 6, DE AGUIAR, E., STOLL, C., THEOBALT, C., AHMED, N., EI- DEL, H.-P. S., AND THRUN, S Performance capure from sparse muli-view video. ACM Trans. Graphics (Proc. SIG- GRAPH), 98. DECARLO, D., AND METAXAS, D The inegraion of opical flow and deformable models wih applicaions o human face shape and moion esimaion. In CVPR, 231. DECARLO, D., AND METAXAS, D Opical flow consrains on deformable models wih applicaions o face racking. In. Journal of Compuer Vision 38, 2, ESSA, I., BASU, S., DARRELL, T., AND PENTLAND, A Modeling, racking and ineracive animaion of faces and heads using inpu from video. In Proceedings of Compuer Animaion, 68. FIALA, M., AND SHU, C Self-idenifying paerns for plane-based camera calibraion. Machine Vision and Applicaions 19, 4, FURUKAWA, Y., AND PONCE, J Dense 3d moion capure for human faces. In CVPR. GUENTER, B., GRIMM, C., WOOD, D., MALVAR, H., AND PIGHIN, F Making faces. In SIGGRAPH 98: Proceedings of he 25h annual conference on Compuer graphics and ineracive echniques, KADIR, T., AND BRADY, M Saliency, scale and image descripion. In. J. Compu. Vision 45, 2, LÉVY, B., PETITJEAN, S., RAY, N., AND MAILLOT, J Leas squares conformal maps for auomaic exure alas generaion. ACM Trans. Graph. 21, 3, LI, H., ROIVAINEN, P., AND FORCHEIMER, R d moion esimaion in model-based facial image coding. IEEE Trans. Paern Anal. Mach. Inell. 15, 6, LIN, I.-C., AND OUHYOUNG, M Mirror mocap: Auomaic and efficien capure of dense 3d facial moion parameers from video. The Visual Compuer 21, 6, MA, W.-C., HAWKINS, T., PEERS, P., CHABERT, C.-F., WEISS, M., AND DEBEVEC, P Rapid acquisiion of specular and diffuse normal maps from polarized spherical gradien illuminaion. In Eurographics Symposium on Rendering, MA, W.-C., JONES, A., CHIANG, J.-Y., HAWKINS, T., FRED- ERIKSEN, S., PEERS, P., VUKOVIC, M., OUHYOUNG, M., AND DEBEVEC, P Facial performance synhesis using deformaion-driven polynomial displacemen maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121. MOHAMMED, U., PRINCE, S. J. D., AND KAUTZ, J Visiolizaion: generaing novel facial images. ACM Trans. Graph. 28, 3, 57. PÉREZ, P., GANGNET, M., AND BLAKE, A Poisson image ediing. ACM Trans. Graph. 22, 3, PIGHIN, F. H., SZELISKI, R., AND SALESIN, D Resynhesizing facial animaion hrough 3d model-based racking. In ICCV, SHEWCHUK, J Triangle: Engineering a 2D Qualiy Mesh Generaor and Delaunay Triangulaor. In Applied Compuaional Geomery: Towards Geomeric Engineering, M. C. Lin and D. Manocha, Eds., vol of Lecure Noes in Compuer Science. Springer-Verlag, WANG, Y., HUANG, X., LEE, C.-S., ZHANG, S., LI, Z., SAMA- RAS, D., METAXAS, D., ELGAMMAL, A., AND HUANG, P High resoluion acquisiion, learning and ransfer of dyanmic 3-d facial expressions. In Compuer Graphics Forum, WILLIAMS, L Performance-driven facial animaion. SIG- GRAPH Compu. Graph. 24, 4, ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M Spaceime faces: high resoluion capure for modeling and animaion. ACM Trans. Graphics (Proc. SIGGRAPH),

10 Figure 18: Capure resuls for one sequence including he reference fooage (op row), pure geomery resul (2nd row), skin srech visualizaion (3rd row), high-qualiy rendering wih exure (4h row), and virual makeup (boom row). Figure 19: Realisic renderings under various differen illuminaions and viewpoins.