3D Hand Pose Reconstruction Using Specialized Mappings

Transcription

1 Boston Unversty Computer Scence Tech. Report No ,Dec (revsed Apr. 2001). To Appear n Proc. IEEE Internatonal Conf. on Computer Vson (ICCV). Canada. Jul D Hand Pose Reconstructon Usng Specalzed Mappngs Rómer Rosales, Vassls Athtsos, Leond Sgal, and Stan Sclaroff Λ Boston Unversty, Computer Scence Department 111 Cummngton St., Boston, MA emal:frrosales,athtsos,lsgal,sclaroffg@bu.edu Abstract A system for recoverng 3D hand pose from monocular color sequences s proposed. The system employs a non-lnear supervsed learnng framework, the specalzed mappngs archtecture (SMA), to map mage features to lkely 3D hand poses. The SMA s fundamental components are a set of specalzed forward mappng functons, and a sngle feedback matchng functon. The forward functons are estmated drectly from tranng data, whch n our case are examples of hand jont confguratons and ther correspondng vsual features. The jont angle data n the tranng set s obtaned va a CyberGlove, a glove wth 22 sensors that montor the angular motons of the palm and fngers. In tranng, the vsual features are generated usng a computer graphcs module that renders the hand from arbtrary vewponts gven the 22 jont angles. The vewpont s encoded by two real values, therefore 24 real values represent a hand pose. We test our system both on synthetc sequences and on sequences taken wth a color camera. The system automatcally detects and tracks both hands of the user, calculates the approprate features, and estmates the 3D hand jont angles and vewpont from those features. Results are encouragng gven the complexty of the task. 1 Introducton The estmaton of hand pose from vsual cues s a key problem n the development of ntutve, non-ntrusve humancomputer nterfaces. The shape and moton of the hand durng a gesture can be used to recognze the gesture and classfy t as a member of a predefned class. The mportance of hand pose estmaton s evdent n other areas as well; e.g.,vdeo codng, vdeo ndexng/retreval, sgn language understandng, computer-aded moton analyss for ergonomcs, etc. In ths paper, we address the problem of recoverng 3D hand pose from a monocular color sequence. Our soluton to ths problem makes use of concepts from stochastc vsual segmentaton, computer graphcs, and non-lnear supervsed learnng. Our contrbuton s an automatc system that tracks the hand and estmates ts 3D confguraton on every frame, that does not mpose any restrctons on the hand shape, does not requre manual ntalzaton, and can easly recover from estmaton errors. 2 Related Work Several exstng systems nclude automated hand detecton and trackng. Such systems typcally make restrctve assumptons on the doman: only hands move, the hands are the fastest movng objects n the scene [17, Λ Ths work was supported n part through Offce of Naval Research Young Investgator Award N , and Natonal Scence Foundaton grants IIS and EIA Fgure 1: Hand pose estmaton overvew. 38, 40, 3, 21, 23], hands are skn colored, or they are the only skn-colored objects n the scene [21, 34]. Often the background s assumed to be statc, and known [21, 8]. Some systems use such assumptons to obtan several possble regons where the hands are, and use matchng wth appearance-based models to choose among those regons [38, 9]. Stochastc tools, such as Kalman flterng [34, 38, 40], can be used to predct the hand poston n a future frame. Overall, hand detecton and trackng algorthms tend to perform well n restrcted envronments, where assumptons about the number, locaton, appearance and moton of hands are vald, and the background s known. Relable performance n more general domans, s stll beyond the current state of the art. Prevous systems representaton of hand pose vares wdely. For certan applcatons, hand trajectores can be suffcent for gesture classfcaton [3, 23]. However, n some domans, knowledge of more detaled hand confguraton must be used to dsambguate between dfferent gestures; e.g.,n sgned languages. Pose can be estmated n 2D or 3D. Most 2D-based approaches try to match the mage of the hand wth vew-based models correspondng to a lmted number of predefned hand poses [9, 4, 38, 11, 17, 35, 34]. In [20] the condensaton algorthm s used to track the ndex and thumb of a hand. Such methods are vald n restrcted domans, n whch users are observed from a known vewpont, performng a lmted varety of motons. One lmtaton n vew-based methods s that pose recog-

2 nton s not vewpont nvarant. Images of the same 3D hand shape from dfferent vewponts, or even rotated examples of the same mage would be consdered dfferent poses. Some of those lmts have been addressed by usng multple cameras [35], and stereo [8]; naturally, such methods wll not work n monocular sequences. Our approach wll avod ths lmtaton through the use of probablstc modelng, Specalzed Mappngs (SMA), to map mage features to lkely 3D hand poses. A related approach to SMA s descrbed n [39], where a system s traned wth vews correspondng to many dfferent hand orentatons and vewponts. Some tranng vews are labeled wth the 3D pose category they correspond to, but most of them are unlabeled. The categores of the unlabeled data are treated as mssng values n a D-EM (Dscrmnant Expectaton-Maxmzaton) framework. The system can recognze 14 hand confguratons, observed from a varety of vewponts. A dfference between that approach and ours s that, n ther system, the confguraton estmaton s formulated as a classfcaton problem, n whch a fnte number of classes are defned. Our SMA approach s based on regresson rather than classfcaton, allowng for theoretcally contnuous solutons of the estmaton problem. Sometmes, such contnuous solutons are preferable to smply recognzng a lmted number of classes. For example, n a vrtual realty applcaton, we may want to accurately reconstruct the hand of the user n the vrtual envronment and estmate the effects of that partcular confguraton on the envronment. Even n cases where the ultmate goal s classfcaton, accurate 3D nformaton can mprove recognton by makng t robust to vewpont varatons. An mportant decson n estmatng 3D pose s the representaton and parameterzaton. Lnk-and-jont models are used by [25, 31], whereas a mesh model s used by [10]. In those three systems, the hand confguraton at the begnnng of a sequence must be known a pror. In addton, self-occlusons and fast motons make t hard to mantan accuracy whle trackng. Our proposed SMA approach avods these drawbacks. SMA s related to machne learnng models [16, 12, 6, 28] that use the prncple of dvde-and-conquer to reduce the complexty of the learnng problem by splttng t nto several smpler ones. In general these algorthms try to ft surfaces to the observed data by (1) splttng the nput space nto several regons, and (2) approxmatng smpler functons to ft the nput-output relatonshp nsde these regons. The splttng process may create a new problem: how to optmally partton the problem such that we obtan several sub-problems that can be solved usng the specfc solver capabltes (.e.,form of mappng functons). In SMA s, we address ths problem by solvng for the parttons and the mappngs smultaneously. In the work of [6], hard splts of the data were used,.e.,the parameters n one regon only depend on the data fallng n that regon. In [16], some of the drawbacks of the hard-splt approach were ponted out (e.g.,ncrease n the varance of the estmator), and an archtecture that uses soft splts of the data, the Herarchcal Mxture of Experts, was descrbed. In ths archtecture, as n [12], at each level of the tree, a gatng network s used to control the nfluence (weght) of the expert unts (mappng functons) to model the data. However, n [12] arbtrary subsets of the experts unts can be chosen. Unlke these archtectures, n SMA s the mappng selecton s done usng a feedback matchng process, currently n a wnner-take-all fashon, but soft splttng s done durng tranng. In applcatons where a feedback map can be computed easly and accurately, ths s an mportant advantage. Also, the shape of the regons that determne ownershp to gven specalzed functons s general; therefore, we do not assume any fxed functonal form or dscrmnant functon to defne these regons (gatng networks). Wth respect to work on learnng based approaches for estmatng artculated body pose, Pont Dstrbuton Models have been appled to recoverng upper-body pose from slhouettes or skn-colored blobs [1, 24]. In [13], a Gaussan probablty model for short human moton sequences was bult. However, ths method assumes that 2D trackng of jonts n the mage s gven. In [2], the manfold of human body confguratons was modeled va a hdden Markov model and learned va entropy mnmzaton. In [33] dynamc programmng s used to calculate the best global labelng of the jont probablty densty functon of the poston and velocty of body features; t was also assumed that t s possble to track these features for pars of frames. These last three approaches model the dynamcs of moton, a problem that n general requres much more tranng data to buld a reasonable approxmaton to the underlyng probablty dstrbuton. 3 Overvew An overvew of our approach can be seen n Fg. 1. Frst s frst traned, gven a number J of example hand jont confguratons are acqured usng a CyberGlove (at approx. 15 Hz). The CyberGlove measures 22 angular DOF of the hand. Computer graphcs software can be used to render a shaded vew of any hand confguraton captured by the CyberGlove. Usng ths computer graphcs renderng functon, we can generate a unform samplng (wth sze S) on the whole vew sphere, and render vews (mages) of every hand confguraton from all sampled vewponts. We can then use mage processng to extract vsual feature fl vector from each of the mages generated; n our case we extract moment based-features, but other features are possble [13]. Ths process yelds a set fψ g =Ψ, where ψ s each of the hand jont confguratons from each vewpont 1, and ffl g =, where fl s a vector of vsual features correspondng to each ψ. These sets Ψ and consttute samples from the nputoutput relatonshp that we wll attempt to learn usng our archtecture. Gven a new mage of a hand, we wll compute ts vsual feature vector x. We then compute the mappng from x to the most lkely 24 DOF hand confguraton. Note that ths mappng s hghly ambguous. In fact the relatonshp s many to many; therefore no sngle functon can perform ths task. Usng the Specalzed Mappng Archtecture (SMA), we splt (partton) ths mappng nto many mappngs. Each of these hopefully smpler problems s then solved usng a dfferent specalzed functon. The SMA learnng scheme solves for parttons and mappngs smultaneously. The SMA tres to learn a multple mappng so that, when performng nference, gven a vector of vsual features x, an output n the output space of hand confguratons can be 1 Ths vector s then composed of 22 nternal pose parameters plus two global orentaton parameters.

3 provded. On the rght column of Fg. 1, a dagram of the nference process s shown. Frst vdeo nput s obtaned, and usng a segmentaton module, regons wth hgh lkelhood of beng skn colored are found. From these regons we extract vsual features (e.g.,moments). Then the gven vector of vsual features x s presented to SMA, whch generates several output estmates, one of whch s chosen usng a defned cost functon. Most of the detals, ncludng the processes of learnng and nference by SMA are presented n the followng sectons. Our approach can easly ntegrate dfferent choces of features. Furthermore, the same approach can be used to estmate the pose of artculated objects other than hands. 4 Hand Shape Representaton The hand model that we use s mplemented n the VrtualHand programmng lbrary [36]. The parameters of the model are 22 jont angles. For the ndex, mddle, rng and pnky fnger, there s an angle for each of the dstal, proxmal and metacarpophalangeal jonts. For the thumb, there s an nner jont angle, an outer jont angle and two angles for the trapezometacarpal jont. There are also abducton angles between the followng pars of successve fngers: ndex/mddle, mddle/rng and rng/pnky. Fnally, there s an angle for the palm arch, an angle measurng wrst flexon and an angle measurng wrst bendng towards the pnky fnger. The VrtualHand lbrary provdes tools that can render an artfcal hand from an arbtrary vewpont, gven values for the 22 angles. Fg. 3 shows examples of hand renderngs. Usng a CyberGlove (manufactured by VrtualTechnologes) we collected about 2,400 examples of hand poses (parameterzed as vectors contanng the 22 angles). We rendered each pose from 86 dfferent vewponts. Those vewponts formed an approxmately unformly dstrbuted set on the surface of a sphere centered at the hand. The synthetc mages obtaned ths way were used for tranng and testng as descrbed n the expermental results. The VrtualHand lbrary was also used to reconstruct the estmated 3D hand shape for testng data, based on the output of our system. 5 Learnng Algorthm The estmaton paradgm used n ths work consst of mappng the observed low-level vsual features to hand jont confguratons. The underlyng approach for fndng ths mappng s based on the Specalzed Mappngs Archtecture (SMA), a non-lnear supervsed learnng archtecture. Gven an nput and output space < c and < t respectvely, SMA s consst of several specalzed forward mappng functons ff k : < c! < t and a feedback matchng functon : < t!< c, whch n ths case s known (vsual features can be obtaned gven the jont confguratons by usng computer graphcs based renderng). In order to estmate these mappngs, we use a supervsed learnng approach wth tranng data Z = fz g =1::n, wth z =(fl ;ψ ) an nput-output par (vsual features and hand jont angles respectvely). Our archtecture generates a seres of m functons ff k n whch each of these functons s specalzed to map certan nputs (ther specalzed doman) better than others. The specalzed doman can be for example a regon of the nput space. However, ths specalzed doman of ff k can be more general than just a connected regon n the nput Fgure 2: SMA dagram llustratng (a) an estmated SMA model wth m specalzed functons mappng subsets of the tranng data (each subset s drawn wth a dfferent color) and (b) the nference process n whch a gven observaton s mapped by all the specalzed functons, and then a feedback matchng step s performed to choose the best of the m estmates. space. We propose to determne these specalzed domans and functons smultaneously. Fg. 2(a) llustrates the basc dea of ths model. We use dfferent colors (gray-levels) to represent the doman of each specalzed functon. At ntalzaton random colors are assgned to each pont, the goal s to fnd an optmal mappng and partton that s effcent n reducng some error functon. Once the model has been learned our mappng may look lke Fg. 2(a), n whch each functon ff s n charge of mappng certan nputs only. 5.1 Probablstc Model Let the tranng sets of output-nput observatons be Ψ = fψ 1 ;ψ 2 ;:::;ψ n g, and = ffl 1 ;fl 2 ;:::;fl n g respectvely. We wll use z = (ψ ;fl ) to defne a gven output-nput tranng par, Z = fz 1 :::z n g represents our observed tranng set. Defne the unobserved random varables y wth = f1::ng and y =(y 1 ;y 2 ;:::;y n ). In our model the varables y have doman the dscrete set C = f1::mg of labels for the specalzed functons, and can be thought as the functon number used to map data pont, therefore m s the number of specalzed functons n the model. Defne the model parameters = ( 1 ; 2 ; ::: m ; ), where represents the parameters of the mappng functon. The vector =( 1 ; 2 ; :::; m ), where k represent P (y = kj ). Usng Bayes rule and assumng ndependence among observatons, we have the jont probablty of the observed and hdden varables condtoned on our model parameters: P (Z; yj ) =P (Zjy; )P (yj ) = Y (a) (b) P (z jy ; )P (y j ) (1)

4 5.2 SMA Parameter Estmaton and the EM Algorthm The optmzaton problem defned by Eq. 1 s computatonally very expensve. Here, the probablstc parameter estmaton problem s approached under the Expectaton Maxmzaton (EM) algorthm framework [5]. We use the notaton followed by [22]. Note that Eq. 1 makes reference to a stll undefned dstrbuton P (z jy ; ). Several optons had been proposed [27]. Here we wll use a Gaussan dstrbuton wth mean defned by the error ncurred n usng the possbly nonlnear functon ff y as a mappng functon, and a varance ± y : P (zjy; ) =N (ψ; ff y (fl; ); ± y ) (2) Usng ths dstrbuton, the E-step conssts of fndng ~P (y) =P (yjz; ). In our case, ths factorzes as: ~P (y) = = Y Y y P (z jy ; ) P k2c kp (z jy = k; ) (3) y e (ψ ff y (fl ; y )) > ± 1 (ψ ff y (fl ; y )) Pk2C ke (ψ ff k (fl ; k ))> ± (4) 1 (ψ ff k (fl ; k )) The M-step conssts of fndng (t) = arg max E P ~ (t) [log P (y; Zj )]. Usng our model, t can be shown that: The update for k depends on the form of ff k. Here we have chosen a non-lnear functon of the form: 5.3 Stochastc Learnng The update equatons descrbed above are useful to fnd a local mnmum gven the ntal values of the parameters. In order to mprove ths process, and avod some of the local mnma that nevtably arse, we use an annealng schedule on the P ~ (t) probabltes durng the M-step. In ths way, we redefne: ~P (t) (y = j) = e log( ~ P (t) (y=j))=t (t) Pk2C elog( ~ P (t) (y=k))=t (t) In our experments the temperature parameter T decays exponentally. Ths step not only does help n avodng local mnma, but t also creates two desrable effects. It forces P ~ (t) (y = j) to be bnary (ether 1 or 0) atlow temperatures, as a consequence each pont wll tend to be mapped by only one specalzed functon at the end of optmzaton. Moreover, t makes P ~ (t) (y = k) (k =1::m)be farly even at hgh temperatures, makng the optmzaton less dependent on ntalzaton. Note that there s no closed form soluton for the M-step as descrbed above. In practce we have decded to perform two or three teratons per M-step. Another source of randomness added to the process so far descrbed conssts n choosng data ponts randomly unformly dstrbuted when performng the M-step. These two varants of the M-step have been justfed n the sense of a partal M-step [22]. 5.4 Feedback Matchng X X Once the model parameters have been estmated each specalzed functon maps (wth dfferent levels of accuracy) (t) = arg max ~P (t) (y )[log P (z jy ; )+log P (y j )]: y 2C the whole nput space. Therefore, the followng queston (5) arses: durng reconstructon, gven a pont n nput space, Ths gves the followng update rules for k and ± k (where Lagrange P how do we choose the mappng functon ff k that should be multplers were used to ncorporate the used to map ths pont? constrant k k =1). Fg. 2(b) llustrates the nference process. When generatng an estmate X ^h of body pose gven an nput x (the k = 1 gray pont wth a dark contour n the lower plane), SMA s P (y = kjz ; ) (6) n generate a seres of output hypotheses H = fhg k obtaned usng h k P P ~ = ff k (x), wth k 2C(llustrated by each of the (t) (y = k)(ψ ff k (fl ; k ))(ψ ff k (fl ; k )) > ponts ponted by the arrows). Gven the set H, we defne ± k = P P ~ (7) (t) (y the most accurate hypothess to be that one that mnmzes = k) the functon F ( (h j ); x; Z), over j, n ths paper we use: (9) = arg mn j ( (h j ) x) > ± 1 ( (h j ) x); (10) fff k (x; k )g q = g 2 ( l 2 X j=0 w (2) qj g 1( l 1 X =0 w (1) j x )); (8) where x s the th component of the vsual feature vector, w (1) and w (2) are weghts and bases (part of k ), g 1 and g 2 are a sgmodal and lnear functon respectvely, l 1 and l 2 are the number of nodes n each layer, and q s just the dmenson ndex of the output vector. Ths s a 1-hdden layer feed-forward network. Unfortunately, usng ths functon (as t would be by usng most non-lnear functons) forces us to use teratve optmzaton for the M-step. and make ^h = h, where ± s the covarance matrx of the elements n the set (.e.,the nput vectors n our tranng set) and s the assgned label. In Fg. 2(b) we can see that each of the ponts n the output space s mapped back to the nput space, once n ths space, these ponts can be compared (usng a gven cost functon e.g.,eq. 10) to the ntal nput observaton. The form of the cost functon could vary, usng Eq. 10 s the same as assummng that P (hjx) =N (h; x; ± ). 6 Hand Detecton and Segmentaton Some of our test data conssts of vdeo sequences collected wth a color dgtal camera. In those sequences the background s statc, there s only one person present, and the

5 person s facng towards the camera. Our system tracks both hands of the user automatcally, usng a skn color tracker. In the frst frame of the sequence, the tracker needs to be ntalzed, by locatng n the mage the objects that we want to track. That could be done by applyng a skn detector system, lke the one descrbed n [15]. However, usng that detector, clothes are labeled as skn, sometmes, because of ther color. We can locate and segment the hands more accurately usng the fact that ther color s very smlar to the color of the face. The poston of the face can be found relably usng a face detector system [29]. For each pxel n the detected face we compute a measure of how skn-lke the pxel color s. That measure s based on hstograms of skn and non-skn color dstrbutons, computed from a database of thousands of mages n whch regons were labeled as skn and non-skn. Those labeled mages were frames from commercally avalable DVD moves. We select the top 50% of the pxels n the face, for whch the measure of skn smlarty s the hghest. For each of R those pxels we compute ts rg color (r = R+G+B ;g = G R+G+B ), and we fnd the mean rg color of all selected pxels. Then, for each pxel n the mage, we calculate the dstance of ts rg color from the mean rg color. We label as skn all pxels for whch that dstance s less than a threshold. The threshold we use s 17, for RGB values between 0 and 255. The objects we want to track are the three largest connected components of the skn pxels. One of them overlaps wth the face, and the other two are consdered to be the hands. We ntalze the skn tracker wth the poston of the face and hand regons, and the tracker locates the face and hands n the rest of the frames n each sequence. The skn tracker models skn color dstrbuton as a hstogram n HSV space. It can handle dstrbutons that change from one frame to the next, because of varyng llumnaton or moton wth respect to lght sources. The changes n skn color that occur n a new frame are modeled as the results of translatng, rotatng and scalng the current hstogram. Furthermore, the evoluton of the hstogram s modeled as a second-order Markov process. The tracker s ntalzed n the frst frame, by beng told whch regons to track, and t estmates the ntal color dstrbuton. In the next 8-30 frames, n addton to trackng and adaptng the skn color hstogram, t also learns the parameters of the Markov process. After the learnng stage, t uses those parameters to predct the color dstrbuton n every new frame, whle stll updatng the Markov model, based on the actual hstogram that s observed n the new frame. The learnng and trackng stage are explaned n detal n [32]. Our smple hand detecton and trackng algorthm would not work at any frame where the hands overlap wth each other or wth the face. In our vdeo sequences we took care to avod such stuatons. Our system could be made more general by ncludng modules to predct occluson of an object by another and to detect when those objects are separated agan. A smlar approach has been successfully appled n the doman of multple person trackng wth occluson handlng [26]. 7 Expermental Results The descrbed approach was tested n experments wth tranng data consstng of approxmately 30 sequences obtaned through the use of a Cyberglove. Input-output pars were generated usng computer graphcs by renderng from 86 vewponts roughly unformly dstrbuted on the vew sphere. The output conssted of 24 jont angles of a human hand lnearly encoded by nne real values usng Prncpal Component Analyss (PCA). The nput conssted of seven real-valued Hu moments [14] computed on synthetcally generated slhouettes of the hand. Hu moments are functons of central mage moments. They are nvarant to translaton, scalng, and rotaton on the mage plane. These nvarances ease the observaton process (e.g.,we do not need to be concern about where and how large the hand appears on the mage). However, rotaton nvarance makes hand rotaton parallel to the mage plane unobservable. For the real experments observaton nputs were obtaned trackng skn color dstrbutons [32]. Approxmately 300,000 mages were generated synthetcally. Of these, 8,000 were used for tranng and the rest for testng. We used cross-valdaton for early stoppng the tranng procedure and avod overfttng. In the experments shown, the number of specalzed functons was set to 30. Each of these functons was a one hdden layer, feedforward network wth 5 hdden neurons. The annealng schedule was 1=k where k was the teraton number n the EM algorthm. Other experments were performed to test the convergence and fttng propertes of the model, due to space lmtatons these results wll not be presented n ths paper. 7.1 Quanttatve Experments Fg. 3 shows example hand confguraton estmates obtaned n representatve test frames (not n the tranng set). Synthetc mages were used n ths experment, because ground-truth data was avalable for quanttatve performance evaluaton. As can be seen n the fgure, selfoccludng confguratons are obvously harder, but stll the estmate s close to ground-truth gven that no human nterventon nor pose ntalzaton was requred. In order to provde quanttatve measures of performance, test data was used to generate vewpont dependent error measures. Fg. 4 shows the mean squared error and ts varance per vewpont at the equator 4(a) and at dfferent lattudes 4(b). Note n 4(a) that for vews on the equator the error s smaller for longtudes closer to ß radans, ths corresponds to a vew of the palm (from dfferent lattudes). These performance dfferences are most lkely due to that at sde-vew angles there s an ncreased amount of self-occluson and also because the projectons nvolve fewer pxels, reducng the samples used to calculate mage moments. In 4(b) we can observe that reconstructon errors ncrease at the poles of the vew sphere, where there s also lttle nformaton projected to the mage plane. Whle the MSE result s encouragng, the varance suggests that certan hand poses are not accurately recovered (we dscover they mostly correspond to complex hand confguratons comng from the Amercan Sgn Language part of our data).

6 MSE n 22 jont confg. space Mean error and varance for all vews on the vewsphere equator Vews (x 2π/16 rads.) MSE n 22 jont confg. space Mean error and varance grouped by lattude Lattude (x 2π/16 rads.) evenly spaced from 90 o to +90 o Fgure 4: Quanttatve expermental results. Mean square error n the reconstructon s shown n (a) taken at the equator of the vew sphere, varyng the longtude and (b) at dfferent lattudes, averagng over all the longtudes. Longtude ß and lattude 0 radans represents a vew towards the palm of the hand. 7.2 Experments wth Real Sequences In the next set of experments, we tested the system aganst real segmented vsual data. The sequences were segmented to yeld blobs that corresponded to hands n each frame, as descrbed n Sec. 6. The resultng reconstructon for several relatvely complex gesture sequences s shown n Fg. 5. Note that gven blob mages, recoverng 3D hand pose s a dffcult task even for a human observer. Ths dffculty s ncreased by performng nference from blob moments, obvously wth an nferor descrptve power. Methods for addressng ths ssue wll be covered further n Sec D Reconstructon Relablty It should be noted that SMA s can provde a measure of reconstructon relablty by usng the log-probabltes computed n Eq. 10. Ambguous nputs can be dscovered by lookng at the relatve scores gven by Eq. 10 (another opton s to look at the entropy of P (hjx) ). Ths s extremely mportant because even though the forward maps are desgned to handle ambgutes, the nference process clearly stll suffers from ambgutes. Therefore, t can be mpossble to recover some confguratons wth enough relablty. As an example, n Fg. 5, the confguratons 5-6 have low relablty score, even though we obtan good estmates. Some of the competng hypotheses nclude estmates that are also consstent (n terms of the vsual features used) wth the nput presented, and some of these consstent hypothess are far from the true 3D reconstructon. Thus, t was very lkely to choose one of the bad estmates nstead of the good ones shown n confguratons Dscusson and Conclusons In ths paper we addressed the problem of recoverng 3D hand pose from a monocular color sequence. The man contrbutons of our work are: 1. A sngle observed frame can be used for estmaton 2. As a consequence, no manual ntalzaton s requred. Furthermore, the sequence can start wth the hand n any poston and orentaton. 2. No lmtaton s mposed n the camera vewponts allowed. 3. The system does regresson rather than classfcaton, thereby provdng a contnuum of pose estmates rather than recognzng a fnte number of classes. 4. A novel non-lnear supervsed learnng framework s adapted to the pose estmaton problem. Ths framework allows us, among other thngs, to avod the ptfalls of explct trackng and to measure relablty of estmates durng nference. 5. Reconstructon can be accomplshed at near frame rate. The man advantage of usng SMA s n ths doman over other functon estmaton paradgms s that t allows modelng of the ambguous nput-output relatonshps that arse. For nstance, dfferent hand confguratons can generate the same vsual features, due to self-occluson. Dfferent vsual features can be related to a sngle hand confguraton, due to naccurate observatons or varatons n hand morphology. SMA s splts (parttons) the problem nto smpler mappng problems. Ths allows for modelng dfferent parts of the output space ndependently, as well as computaton of multple possble confguratons n ambguous stuatons. However, so far we choose one estmate only. Ths s an nterestng aspect not fully addressed n ths paper, Sec. 7.3 extends a lttle on ths topc. In our current mplementaton, temporal context s not used for mprovng the output estmates durng mappng, but only for segmentaton. The hand pose s re-estmated at every frame gven the segmented data. We expect that usng prevous estmates n computng the current hand shape wll mprove accuracy, and we plan to extend our approach to allow ths. However frame ndependence allows a very attractve nference tme of O(M ), wth M specalzed functons. Our algorthm could be used as a front end n several gesture recognton applcatons that take the hand confguraton as nput. Current systems rely almost exclusvely on non-vson technques to obtan such data, such as CyberGloves [7, 18, 19, 30] and color markers [11]. An automated computer vson technque lke ours mposes no restrctons on users. It can also be used n domans where we do not have control of the data collecton, and therefore we cannot requre the use of more sophstcated nput devces. In future work, we plan to experment wth sets of features that are rcher and more descrptve than bnary slhouettes; e.g.,orentaton hstograms, or other texture features. Usng stereo should further ncrease the accuracy of the system, by provdng more shape constrants than a sngle 2D mage does. Fnally, more sophstcated models of temporal dependences, lke lnear Gaussan Models n general [11, 34, 37], could be used n the feedback matchng to gude the choce of best reconstructon. Even though we have a useful estmate of confdence, gven by Eq. 10, we are lookng at alternatves for decreasng the error varance. 3D Hand pose reconstructon from a sngle mage s a very dffcult task, and at present no fully-general soluton to the problem exsts. Our results show that t s possble to approach ths problem usng a combnaton of vson and statstcal learnng tools. We consder ths an mportant step consderng the complexty of the task and the low descrptve power of the features currently employed. 2 We nsst that n applcatons where hghly correlated frames can be observed, t s mperatve to use ths temporal nformaton. However, the ablty of our framework to estmate hand pose gven only a sngle frame affords automatc ntalzaton, faster estmaton, and could be used as a bootstrap mechansm n more complex systems.

7 References [1] R. Bowden, T. Mtchell, and M.Sarhad. Non-lnear statstcal models for the 3d reconstructon of human body pose and moton from monocular mage sequences. Image Vson Comp., 18(9: ), [2] M. Brand. Shadow puppetry. In ICCV, [3] R. Cutler and M. Turk. Vew-based nterpretaton of realtme optcal flow for gesture recognton. In Face and Gesture Recognton, pages , [4] T.J. Darrell, I.A. Essa, and A.P. Pentland. Task-specfc gesture analyss n real-tme usng nterpolated vews. PAMI, 18(12), [5] A. Dempster, N. Lard, and D. Rubn. Maxmum lkelhood estmaton from ncomplete data. Journal of the Royal Statstcal Socety (B), 39(1), [6] J.H. Fredman. Multvatate adaptve regresson splnes. The Annals of Statstcs, 19,1-141, [7] M. Fröhlch and I. Wachsmuth. Gesture recognton of the upper lmbs : From sgnal to symbol. In I. Wachsmuth and M. Fröhlch, edtors, Gesture and Sgn Language n Human- Computer Interacton, Gesture Workshop, pages , Belefeld, Germany, [8] R. Grzeszczuk, G. Bradsk, M.H. Chu, and Jean-Yves Bouguet. Stereo based gesture recognton nvarant to 3d pose and lghtng. In CVPR, volume 1, pages , [9] R. Hamdan, F. Hetz, and L. Thoraval. Gesture localzaton and recognton usng probablstc vsual learnng. In CVPR, volume 2, pages , [10] T. Heap and D. Hogg. Towards 3d hand trackng usng a deformable model. In Face and Gesture Recognton, pages , [11] H. Henz, K. Krass, and B. Bauer. Contnuous sgn language recognton usng hdden markov models. In Intl. Conf. on Multmodal Interfaces, volume 4, pages 10 15, [12] G. Hnton, B. Sallans, and Z. Ghahraman. A herarchcal communty of experts. Learnng n Graphcal Models, M. Jordan (edtor), [13] N. Howe, M. Leventon, and B. Freeman. Bayesan reconstructon of 3d human moton from sngle-camera vdeo. In NIPS, [14] M. K. Hu. Vsual pattern recognton by moment nvarants. IRE Trans. Inform. Theory, IT(8), [15] M.J. Jones and J.M. Rehg. Statstcal color models wth applcaton to skn detecton. In CVPR, pages I: , [16] M. I. Jordan and R. A. Jacobs. Herarchcal mxtures of experts and the EM algorthm. Neural Computaton, 6, , [17] M. Kohler. Specal topcs of gesture recognton appled n ntellgent home envronments. In Proceedngs of the Gesture Workshop, pages , [18] R. Lang and M. Ouhyoung. A real-tme contnuous gesture recognton system for sgn language. In Face and Gesture Recognton, pages , [19] J. Ma, W. Gao, and C. Wang J. Wu. A contnuous chnese sgn language recognton system. In Face and Gesture Recognton, pages , [20] J.P. MacCormck and M. Isard. Parttoned samplng, artculated objects, and nterface-qualty hand trackng. In ECCV, [21] J. Martn, V. Devn,, and J.L. Crowley. Actve hand trackng. In Face and Gesture Recognton, pages , [22] R. Neal and G. Hnton. A vew of the em algorthm that justfes ncremental, sparse, and other varants. Learnng n Graphcal Models, M. Jordan (edtor), [23] A. Nshkawa, A. Ohnsh, and F. Myazak. Descrpton and recognton of human gestures based on the transton of curvature from moton mages. In Face and Gesture Recognton, pages , [24] E-J. Ong and S. Gong. Trackng hybrd 2d-3d human models through multple vews. In ICCV Workshop on Modellng People, Corfu, Greece, [25] J.M. Rehg. Vsual Analyss of Hgh DOF Artculated Objects wth Applcaton to Hand Trackng. PhD thess, Electrcal and Computer Eng., Carnege Mellon Unversty, [26] R. Rosales and S. Sclaroff. Improved trackng of multple humans wth trajectory predcton and occluson modelng. In IEEE CVPR Workshop on the Interpretaton of Vsual Moton, [27] R. Rosales and S. Sclaroff. Specalzed mappngs and the estmaton of body pose from a sngle mage. In IEEE Human Moton Workshop. Austn, TX, [28] R. Rosales and Stan Sclaroff. Inferrng body pose wthout trackng body parts. In CVPR, [29] H.A. Rowley, S. Baluja, and T. Kanade. Rotaton nvarant neural network-based face detecton. In CVPR, pages 38 44, [30] H. Sagawa and M. Takeuch. A method for recognzng a sequence of sgn language words represented n a japanese sgn language sentence. In Face and Gesture Recognton, pages , [31] N. Shmada, Y. Shra, Y. Kuno, and J. Mura. Hand gesture estmaton and model refnement usng monocular camera - ambguty lmtaton by nequalty constrants. In Face and Gesture Recognton, pages , [32] L. Sgal, S. Sclaroff, and V. Athtsos. Estmaton and predcton of evolvng color dstrbutons for skn segmentaton under varyng llumnaton. In CVPR, [33] Y. Song, Xaolng Feng, and P. Perona. Towards detecton of human moton. In CVPR, [34] T. Starner and A. Pentland. Real-tme amercan sgn language recognton usng desk and wearable computer based vdeo. PAMI, 20(12): , [35] A. Utsum and J. Ohya. Multple-hand-gesture trackng usng multple cameras. In CVPR, volume 1, pages , [36] Vrtual Technologes, Inc., Palo Alto, CA. VrtualHand Software Lbrary Reference Manual, August [37] C. Vogler and D. Metaxas. Toward scalablty n asl recognton: Breakng down sgns nto phonemes. In Proceedngs of the Gesture Workshop, [38] J. Weng and Y. Cu. Recognton of hand sgns from complex backgrounds. In R. Cpolla and A. Pentland, edtors, Computer Vson for Human-Machne Interacton. Cambrdge Unversty Press, [39] Y. Wu and T.S. Huang. Vew-ndependent recognton of hand postures. In CVPR, volume 2, pages 88 94, [40] M. Yang and N. Ahuja. Recognzng hand gesture usng moton trajectores. In CVPR, volume 1, pages , 1999.

8 (a) (b) (a) (b) (a) (b) Fgure 3: Example reconstructon of several synthetc test sequences. Each set (2 rows each) conssts of (a)nput mages, (b)reconstructon. Because our approach can provde us wth a reconstructon confdence, we used ths to show hgh-medum-low confdence estmates (one par of rows each). Fgure 5: Reconstructon obtaned from performng hand segmentaton n a human subject. The two top pars of rows show good reconstructon whle the bottom par show examples of bad performance. Reconstructon s shown from a fxed vewpont (lattude 0-longtude ß rads.).