3D Head Tracking Based on Recognition and Interpolation Using a Time-Of- Flight Depth Sensor

Transcription

1 3D Head Tacng Based on Recognton and Intepolaton Usng a Tme-Of- Flght Depth Senso Salh Bua Götü 1 and Calo Tomas 1,2 1 Canesta Inc., 2 Due Unvesty bgotu@canesta.co tomas@cs.due.edu Abstact Ths pape descbes a head-tacng algothm that s based on ecognton and coelaton-based weghted ntepolaton. The nput s a sequence of 3D depth mages geneated by a novel tme-of-flght depth senso. These ae pocessed to segment the bacgound and foegound, and the latte s used as the nput to the head tacng algoth whch s composed of thee mao modules: Fst, a depth sgnatue s ceated out of the depth mages. Next, the sgnatue s compaed aganst sgnatues that ae collected n a tanng set of depth mages. Fnally, a coelaton metc s calculated between most possble sgnatue hts. The head locaton s calculated by ntepolatng among stoed depth values, usng the coelaton metcs as the weghts. Ths combnaton of depth sensng and ecognton-based head tacng povdes moe than 90 pecent success. Even f the tac s tempoaly lost, t s easly ecoveed when a good match s obtaned fom the tanng set. The use of depth mages and ecognton-based head tacng acheves obust eal-tme tacng esults unde exteme condtons such as 180-degee otaton, tempoay occlusons, and complex bacgounds. 1. Intoducton Head tacng s a ey component n applcatons such as human compute nteacton, peson montong, dve montong, vdeo confeencng, and obect-based compesson. All of these applcatons eque obust, and eal-tme (o nea eal tme) tacng. On the othe hand, head tacng algothms ae nown to lose tac dung an abupt movement, o dung otaton of the face n a vdeo sequence. A pactcal soluton should expect the undelyng tacng algothm to fal, and should povde easy ecovey. In ths pape, we addess the tacng poblem fo a peson sttng n font of the camea. The bacgound can potentally be a clutteed bacgound. The peson s fee to otate hs o he head and body as much as 180 degees, to patally move out of the pctue, o to patally occlude hs o he head. We am to povde a obust soluton that coves all possble abovementoned condtons, whle etanng a epoecton accuacy of wthn a few pxels. Ou system uses a tme-of-flght based depth senso to geneate the nput mages. Depth mages ae sutable to ou poblem n vaous ways. Fst, depth mages povde geomety nfomaton dectly, as opposed to the photometc nfomaton contaned n ntensty mages. Fo example, the clutteed bacgound can be elmnated va depth segmentaton. Next, fndng the head s locaton n the mage also povdes depth, and theefoe chaactezes the head s 3D locaton fully. Thd, the depth mage s not affected much by the textue on a face o envonmental llumnaton condtons, and the tace does not get out of tac even wth full otatons of the head and body. Ou ecognton-based head tacng algothm nvolves a tanng and a testng stage. Dung tanng, nowledge-based clusteng s appled to constuct clustes of sgnatues fo a dense samplng of locatons of the head n the depth mage. Each locaton (ncludng z values) s used as the coespondng label n a supevsed clusteng algothm. Ths assues that we eep evey ndvdual confguaton n a patcula locaton whle stll compessng the tanng data fo effcency puposes. In the testng stage, the sgnatue of a test mage s compaed aganst the sgnatues stoed wth the tanng clustes. Once all satsfactoy matches ae detemned, coelaton values between new and stoed depth mages ae computed aound the possble head locaton to geneate match-qualty metcs. The fnal stage of the algothm nvolves ntepolaton weghted by the dstance of the depthsgnatues and these match-qualty coelaton metcs. Ths stage s necessay snce thee ae a lage numbe of confguatons, and ntal hts wth the tanng set mght be msleadng. Intepolaton s especally useful

2 when a good match n the tanng set does not exst, whle matches aound the good match do exst. Thee ae vaous othe tacng and head tacng algothms n the lteatue. When a pont-featue tacng algoth such as the Lucas-Kanade tace [1,2,3] s appled to faces, t seems to be easly confused when the textue of the face changes as a esult of tanslaton o otaton. Model-based head taces [4,5,6,7,8,9] ae theefoe pefeable. Thee have been dffeent choces of head models n the lteatue. Complex model-based methods usually eque a sepaate head model fo each peson. In othe wods, these methods ae not sutable fo applcatons whee one model needs to ft any peson. Some eseaches have epoted that the use of moe smplstc methods esulted n obust tacng. Fo example, Stan Bchfeld s [10] head tacng algothm shows good esults, yet the system seems to fal wth complex bacgounds. In addton, hs optmzaton s based on full seach, whch lmts ts use n eal tme applcatons. Vaous eseaches have appled tacng based on depth mages. Yang and Zhang [11] have appled head tacng usng steeovson. Ths wo s stll dependent on the bghtness nfomaton due to the natue of steeo magng, and thus t s senstve to clutteed bacgounds o llumnaton condtons. Malassots and Stntzs [12] have ecently poposed a head tacng algothm based on ange mages obtaned usng colo coded stuctued lght. The wo models the mages usng a Gaussan mxtue of head and toso. Snce the am o hands have not been modeled, ths wo would fal unde challengng confguatons. Recently, vaous eseaches llustated the use of ecognton and ntepolaton famewo fo tacng. Tomas [13] descbed a tacng technque based on classfcaton followed by ntepolaton, wth applcaton to hand postue tacng. Smlaly, Naya [14] has poposed an appeaance-based ecognton scheme fo contollng a obot am va tacng. He uses PCA, and ths mght esult n ulng out confguatons that do not occu fequently. Naya s and Tomas s appeaance-based ecognton appoach povdes seveal advantages fo the tacng tas. Fst, posng the ecognton metc n the mage space athe than a deved epesentaton s vey poweful, snce the data may povde a bette epesentaton than abstactons fo many cases. At the same tme, appeaance-based ecognton has vaous dsadvantages as well. Segmentaton and obect detecton s stll an ssue. In addton, t has lmted powe fo ntepolaton and to genealze to novel condtons. Ths pape dffeentates tself fom pevous wo n vaous means. Fst, novel depth sensng hadwae s used to geneate the nput mages. Use of depth mages povdes vaous benefts such as nvaance to clutteed bacgound, textue of the face, and oentaton of the head. Thee s a vast amount of eseach n 3D obect tacng and 3D obect sgnatues. Ou wo combnes these contbutons wth a ecognton based famewo, and s unque n ts data-dven appoach. The use of ecognton based tacng has vaous advantages. Fst of all, thee s no need to geneate an atculated head/face model no to chaacteze ts paametes. The model s automatcally leant n the ecognton famewo. We also dffeentate ouselves fom moe ecent appeaance and ecognton based tacng wo. Fst, ou depth mages assue that the foegound obects can be segmented fom the bacgound, theeby solvng the obect segmentaton and detecton poblems. We povde a scheme to obtan compessed featues fom depth mages. Next, we apply a nowledge-based ecognton famewo based on clusteng of labeled data. Ths assues that no mpotant confguaton s mssed. Ou ntepolaton scheme uses a coelaton metc, assung that t does not get dstacted by msmatches. Anothe advantage s the obustness of the poposed famewo. Smlaly to othe tacng algothms, the tac mght tempoaly be lost, yet t s ecoveed qucly snce the famewo s dependent on fndng a good match n the tanng set, athe than pefomng a local seach n paamete space. The pape contnues as follows: Fst, we gve an ovevew of the wong pncple of the tme-of-flght senso. Next, we dscuss ou ecognton-based headtacng algothm. In the followng sectons, we descbe the depth-based sgnatues, nowledge-based clusteng algoth sgnatue matchng, and coelaton based ntepolaton espectvely. Fnally, we show expements and esults, and close wth ou conclusons and dscussons. 2. Tme-Of-Flght Depth Senso The Canesta tme-of-flght magng system was used n ou expements to poduce the depth map. The system conssts of a modulated lght souce such as a lase, a CMOS senso consstng of an aay of pxels each capable of detectng the ntensty and phase of the ncomng lght, and an optcal system fo focusng. Dstance s computed fom the phase of the modulaton envelope of tansmtted nfaed lght as eceved at a pxel. Let s(t)=sn(2 f m t) be the tansmtted lght whee f m s the modulaton fequency. The lght s eflected fom a taget, and etuns bac to a senso pxel wth a phase shft :

3 2d ( t) Rsn(2f m t-) Rsn(2f m ( t- )) c whee R s the ampltude of the eflected lght, d s the dstance between the senso and the taget and c s the speed of lght, 3x10 8 m/s. The dstance d can be calculated fom the phase shft as follows: Senso Lght souce Lens Taget c d 4f m A depth mage s constucted by measung the dstance d at evey pxel. Smlaly, a bghtness mage s constucted va measung R at evey pxel. The phase detecton was mplemented n CMOS ccuty as descbed n [15, 16]. The tme of flght senso s dffeent fom othe depth senso n vaous ways. Fst, unle steeo, t s textue ndependent. The amount of post-pocessng s mnmal o none, gvng applcaton-pocessng moe tme fo eal tme opeablty, and vey fast fame ates can be obtaned f needed. It uses flood lght, as opposed to stuctued lght, and the system can be tggeed even wth a small amount of lght. Ths povdes a compaable advantage ove stuctued lght systems, snce thee s no movng lght pat and no esultng eye-safety poblem. The system does not necesstate a baselne between the lght souce and the camea, and as such thee s no paallax shadows. Fnally, the depth senso s mplemented on a CMOS chp, and ths povdes an nexpensve and elatvely hgh-esoluton depth senso fo compute vson. An example depth mage of a peson s shown n Fgue 2. Hee, we colo code the mages such that the bacgound s unfom n colo, and foegound pxels ae dae whee the scene s close to the camea. The output depth mages ae used as nput vdeo sequences to ou tacng algothm as descbed next. 3. Ovevew of the Recognton-Based Tacng Algothm The algothm has two man stages, a tanng stage, and a testng o tacng stage. In the tanng stage, long sequences of moves ae captued fom multple people. A depth-sgnatue s calculated on each fame as descbed n the followng secton. In addton, a wndow aound the head locaton s manually dentfed on each fame. The tanng algothm conssts of a nowledge-based clusteng algothm. The esultng output s a set of good epesentatve sgnatues fo possble head locatons. d Fgue 1. Modulated lght eflects bac fom the tagets, and the tme of flght s used to measue the depth d. Fgue 2. An example depth mage. The pxels become dae as the obects ae close to the camea. Fgue 3. Image llustatng the and c vectos obtaned out of an mage. In the testng o tacng stage, each fame s pocessed to obtan ts depth sgnatue. Fst, the sgnatues n the tanng set ae compaed aganst the sgnatue of the test fame. Ths povdes a numbe of most lely matches. Next, a coelaton metc s calculated between these and the test mage. Fnally, locaton of the head s detemned va ntepolaton between the most lely matches usng the coelaton metc. Detals ae descbed next.

4 4. Depth Sgnatues Each depth mage s pocessed to ceate a depth sgnatue. Snce the sgnatues ae late used fo tacng the poston of the head, we need featues that ae senstve to shape vaatons and tanslatons. Specfcally, a 134-dmensonal featue vecto, f fo each mage I s ceated: f m w c T whee m and ae the mean and standad devaton of the depth values n I, w s a vecto that contans the uppe left and bottom ght locatons of the wndow aound the head, and and c ae two vectos whee the th values and c ae obtaned as follows: I (, ) c K I (, ).,, I (, ) 0, I (, ) 0 Hee, I ( s the value of the depth mage at ow m and column n, and K s a constant. In othe wods, and c ae modfed ow and column sums of the mage I. These vectos ae llustated n Fgue 3. The components and c compess the whole depth mage nto a vecto of length equvalent to sum of the numbe of ows and columns. Whle dong so, t stll eeps the elevant nfomaton n the depth mage. 5. Tanng va Knowledge-Based Clusteng Cluste-based leanng s appled to the depth sgnatues. One possblty s to apply clusteng o leanng dectly on the tanng set. K-means clusteng o pncpal component analyss can be appled fo ths pupose. Howeve, these appoaches ae dvesely affected by the numbe of occuences of cases. Fo nstance, pncpal component analyss would teat a case that happens vey nfequently as nose. Instead, we apply clusteng only afte we dvde the sgnatues nto bucets of possble head locatons. We use a modfed vecto quantzaton fo ths pupose. Fst, the mage s dvded nto wndows. Next, the sgnatue of each tanng mage s assgned nto the wndow whee the head s located n that mage. Afte ths assgnment, each wndow contans a numbe of sgnatues. Fnally, sgnatues that fall n each wndow ae clusteed usng -means clusteng. We apply an teatve -means algothm [17] whee we stat fom a elatvely lage, and teate untl each cluste has suffcent numbe of elements. At each teaton, we decease by elmnatng the clustes wth nsuffcent numbe of elements. The sgnatues that ae closest to the fnal cluste centes and the mages assocated wth them ae ept as the epesentatve depth sgnatues and epesentatve mages to be used n sgnatue matchng and tacng. 6. Sgnatue Matchng and Coelaton Based Intepolaton Tacng conssts of sgnatue matchng followed by coelaton-based ntepolaton. Fst, the sgnatue of the nput depth mage s obtaned and compaed wth the sgnatues n the tanng set that eman afte nowledge-based clusteng. Ths povdes best matches. It s possble that thee ae completely dffeent confguatons wthn these best matches. To avod gettng affected by bad matches, we apply coelaton between the possble matches and the test mage. Fnally, we ntepolate between the emanng matches to obtan the fnal head locaton. The nput s the depth mage of the tacng scene. Fst, a sgnatue of the nput depth mage s obtaned as descbed n Secton 4. Next, the sgnatue s compaed aganst the epesentatve sgnatues that wee obtaned usng nowledge-based clusteng. Let f and f be the sgnatues of the tacng mage and a cluste cente espectvely: f m w c f m w T c T The wndow of the pevous taced-fame s used as wndow w. The dstance D between f and f s ceated as follows: D K m m K K w w K K sgn( sgn( c M ( )) sgn( M ( c )) sgn( c 3 M ( )) M ( c )) whee sgn() and M() ae sgn and medan opeatons espectvely, and K 1...K 5 ae constants. Typcal values of these constants ae 0.1, 0.1, 0.1, 1, 1 espectvely. Once the dstance D s obtaned fo evey cluste cente n the tanng set, the most lely detectons ae obtaned as the cluste centes wth a small dstance D. Examples of possble detectons ae gven n Fgue 4 wth ectangles wth thn edges. The possble detectons ae late evaluated by a coelaton metc as descbed next. Each possble locaton suggests a wndow w fo the locaton of the head. Let I be the taced depth mage and I be the mage coespondng to a possble match. Fst I and I ae nomalzed aound the wndow w to

5 obtan the nomalzed depth mage patches N and N espectvely: I ( I ( N( N ( 1 1 nw I(, l) nw I (, l) W W, lw, lw whee W s the numbe of elements n wndow w. The coelaton metc C between the nomalzed mage patches N and N s obtaned as follows: C N ( N ( n Once the coelaton metc fo each possble detecton =1 n s detemned, the locaton of the head locaton (w) s detemned by an ntepolaton scheme gven as follows: w n 1 n K K 6 D C 7 1 D K6 C K7 w whee K 6 and K 7 ae constants. Typcal values ae 1 to 5. In othe wods, the fnal detecton w s an ntepolated wndow between the possble matches. The ntepolaton s useful, snce the tanng set does not necessaly cove all possble head locatons, and ntepolaton assues that n-between confguatons can be constucted. 7. Expements and Results We have conducted expements on a set of 10 sequences wth 8 people, fo a total of 2287 depth fames. We used the Canesta tme-of-flght depth senso, whch povded 64x64 depth mages at a fame ate of 30 fps. Next, we constucted a tanng set fom the set of moves. We obtaned sgnatues fo each fame n the moves, whch we clusteed nto 1443 fames. These ncluded cases at 274 dffeent head locatons n the mage. Then, we constucted a testng set wth moves fom sx people that also had moves n the tanng set, and two people that dd not have any moves n the tanng set. We manually chose the head locaton n each mage so that we could compae the expement esults wth the manually selected gound tuth. Fgue 5 shows the benefts of ntepolaton: possble head locatons ae dentfed by the thn lnes, and the esultant head locaton by the thc lne. Ths coects satsfactoly fo wong ntal matches, and fo cases whee the exact locaton does not exst n the tanng set. Fgue 4. Example of Possble Detectons Next, we povde examples of tacng fames fom the thee test moves n Fgue 6, 7, 8 and 9 espectvely. As one can see, thee ae vaous challengng confguatons of the head. The system s able to tac the head n these challengng confguatons, such as the hand s n the mage (Fgue 6, and 8), o such as the head s patally occluded by the hand (Fgue 8), o the head was baely vsble due to lac of lght (Fgue 5) o whee only a poton of the head s vsble n the mage (Fgue 6 and 7), o whee the head made exteme otatons (Fgue 6 and 7). Sometmes, the head mght be tempoaly out of tac when the head s out of pctue o t s totally occluded. The system easly gets bac nto tac when a good match s found between the tanng set and the test mages. To evaluate the pefomance of the algoth we calculate the aea-wse ovelap wth the algothm detecton and the manually pced gound tuth. We povde a cuve fo test moves whee the subect was also pat of ou tanng set (Fgue 10) and fo test moves whee the subect was not (Fgue 11). The x axs s the amount of aea-wse ovelap, and the y-axs s the facton that had at least that pecentage of ovelap (e.g. ths s analogous to a cumulatve pobablty functo. On aveage, 60 pecent ovelap coesponded to a msmatch of 2 pxels n x and y on the head cente. An 80 pecent ovelap coesponded a msmatch of 1 pxel n x and y on the head cente. We obseve obust esults such that thee s at least 70 (60) pecent ovelap on 85 (95) pecent of the fames fo the cases whee the subect was also ncluded n the tanng set (Fgue 10). We also obseve that thee s at least 70 (60) pecent ovelap on 80 (90) pecent of the fames fo the cases whee the subect was not ncluded n the tanng set (Fgue 11). All the pocessng was executed n eal tme usng Mcosoft Vsual C/C++ on a Pentum 3 Pocesso.

6 8. Conclusons Ths pape llustated the use of a tme-of-flght depth senso fo a head tacng applcaton. In ou poble we amed to obtan a obust head-tacng algothm that would tac challengng cases such as vaous confguatons that nvolved the head and the hand, patal occlusons, patal out of mage postons. Ou algothm apples ecognton followed by ntepolaton. We use a nowledge-based tanng algoth whee the data s fst dvded nto ntal clustes dependng on whee the head s located at the patcula fame. Then, a modfed -means clusteng s appled to cluste the data n the ntal cluste. Knowledge-based clusteng assues that all possble cases ae accounted fo n the tanng set. Whle testng a new fame, we fst fnd a set of possble matches fom the tanng set. Due to the complexty of the head/body confguatons, t s possble that thee ae wong matches. We calculate a coelaton metc between the possble matches and the test mage, fo use n the fnal ntepolaton. The pape maes vaous contbutons. Fst, we suggest usng the depth mages fo obust headtacng, snce t s ease to nfe geomety and 3D locaton fom depth mages. Next, we povde a fast ecognton and ntepolaton based head tacng algothm. Ths algothm s obust n vaous confguatons, snce t neve gets nto a local seach deadloc when the tac s tempoaly lost. The poposed sgnatues wo best fo depth mages, although the geneal famewo of ecognton based tacng s applcable to ntensty mages wth the appopate choce of sgnatues. 10. Refeences [1] B.D. Lucas, and T. Kanade, An teatve mage egstaton technque wth an applcaton to steeo vson,poc. 7th IJCAI, 1981, [2] J. Sh, C. Tomas, Good Featues To Tac, CVPR, 1994, [3] J. Baon, D. Fleet, D. and S. Beauchemn, Pefomance of optcal flow technques, IJCV, 1994, [4] D. DeCalo, and D. Metaxas, Defomable model-based face shape and moton estmaton, Poc. 2 nd Int.l Conf. on Automatc Face and Gestue Recognton, 1996, [5] D. DeCalo, and D. Metaxas, The Integaton of Optcal Flow and Defomable Models wth Applcatons to Human Face Shape and Moton Estmaton, CVPR, 1996, [6] P. Eset, and B. God, Model-based Facal Expesson Paametes fom Image Sequences, Poc. IEEE Intenatonal Confeence on Image Pocessng, Santa Babaa, CA, USA, 1997, [7] P. Eset, and B. God, Analyzng Facal Expessons fo Vtual Confeencng, IEEE Comp. Gaphcs and Appl., Compute Anmaton fo Vtual Humans, 1998, [8] S.B. Gotu, J.Y. Bouguet, R. Gzeszczu, A data dven model fo monocula face tacng, ICCV, 2001, [9] S.B. Gotu, J.Y. Bouguet, C. Tomas, B. God, Model-Based Face tacng fo Vew-Independent Facal Expesson Recognton, IEEE In.l Conf. on Face and Gestue Recognton, 2002, [10] S. Bchfeld, Ellptcal Head Tacng Usng Intensty Gadents and Colo Hstogams, CVPR, 1998, [11] R. Yang and Z. Zhang: "Model-Based Head Pose Tacng Usng Steeovson", IEEE In.l Conf. on Face and Gestue Recognton, 2002, [12] S.Malassots and M.G.Stntzs: "Real-tme Head Tacng and 3D Pose Estmaton fom Range Data", ICIP 2003, [13] C. Tomas, S. Petov and A. Sasty. 3D tacng = classfcaton + ntepolaton. ICCV, 2003, [14] S. K. Naya, S. A. Nene, and H. Muase, Subspace Methods fo Robot Vson, IEEE Tans RA, 1996, 12(5) [15] C. Ba CMOS Compatble 3-D Image Senso, U.S. Patent No: 6,323,942. [16] C. Ba E. Chabon, CMOS Compatble Thee- Dmensonal Image Sensng Usng Reduced Pea Enegy, U.S. Patent No: 6,580,496. [17] S.B.Gotu, Shape Recognton wth Applcaton to Medcal Imagng, PhD Thess, Stanfod Unvesty, Fgue 5. Examples of ntepolaton. Coelaton based metc helps avodng dstacton by ntal msmatches. Fgue 6. Tacng esults fom a move. Fames 1, 17, 22, 27, 58, 93. The head s successfully taced n vaous dstance and locaton of the head.

7 Fgue 7. Tacng esults on anothe move. Fames 1, 4, 18, 44, 53, 79. The head s taced n nstances whee the head s mostly out of pctue, o whee the hand s n the pctue, povdng confusng confguatons. Fgue 10. Pefomance cuves fom test moves whee the subect was also patcpatng n tanng moves. Each cuve coesponds to anothe test move. Fgue 8. Tacng esults on anothe move. Fames 8, 15, 50, 63, 92, 132. The head s taced unde exteme otatons. Fgue 11. Pefomance cuves esultng fom test moves whee the subect was not patcpatng n tanng moves. Each cuve coesponds to anothe test move. Fgue 9. Tacng esults on anothe move. The head s patally occluded by the hand on these fames.