Intuitive Interactive HumanCharacter Posing with Millions

Transcription

1 Feature Artcle Intutve Interactve HumanCharacter Posng wth Mllons of Example Poses Xaoln K. We and Jnxang Cha exas A&M Unversty A n nteractve system that lets users qucly pose a 3D full-body human character has many applcatons for example rapd prototypng of 3D human anmaton real-tme game character control nteractve humanod robot manpulaton and human-computer nteractons. Buldng such a system presents two major challenges. Frst typcal user nput nformaton for example pont draggng s qute low-dmensonal compared to a typcal humancharacter model whch s often A data-drven algorthm represented by more than 50 for nteractve posng of degrees of freedom. he low3d human characters uses dmensonal nputs can t coma maxmum a posteror pletely determne a full-body framewor that combnes human pose because they re conuser nputs wth pose prors sstent wth many dsparate solutons some of whch mght not learned from a large and correspond to natural poses or heterogeneous moton the user s ntentons. database. Intutve system he second challenge s pronterfaces smplfy pose vdng an easy-to-use nterface manpulaton and setchng. for 3D character posng. Such he system performs better nterfaces requre a system that than standard nversecan support varous user connematc and alternatve strants to accommodate users data-drven technques. wth dfferent sll levels. One appealng soluton to reducng reconstructon ambguty s to mpose pror nformaton from several prerecorded human poses. Systems often constran the soluton space by usng weghted combnatons of pose examples or acton-specfc statstcal pose models (see the Related Wor sdebar). hese systems wor well for a small amount of tranng data but 78 July/August 0 ther performance deterorates rapdly as the pose data s sze and heterogenety ncrease. hs sgnfcantly lmts ther utlty because current motoncapture databases are becomng ever larger and more heterogeneous. For example the Carnege Mellon Unversty moton-capture (CMU mocap) onlne database ( contans about three mllon poses from more than 40 dfferent human actons. We ve addressed ths challenge by developng a way to effcently model pror nformaton from a huge heterogeneous database n our case the CMU mocap database and use the models for nteractve human-character posng. We ve also developed two ntutve nterfaces for real-tme character posng both sutable for novce users. We ve demonstrated our approach s flexblty and generalzablty by nteractvely posng 3D characters based on random photos downloaded from the Internet. A comparson of our system wth prevous methods shows our algorthm s superorty. Constranng User Input We base our approach on extractng pror-pose nformaton embedded n the CMU mocap database and usng the nformaton to generate a desred pose from varous forms of user-specfed nematc constrants. We formulate the nteractve character-posng problem n a maxmum a posteror (MAP) framewor. Maxmzng the posteror that s the lelhood of character poses gven the user-specfed constrants produces a most-lely human pose that acheves the user s specfed goal. Wthn the MAP framewor we estmate a from the user s nput c: most-lely pose q Publshed by the IEEE Computer Socety 07-76//$ IEEE

2 Related Wor n Character Posng I nverse nematcs (IK) s one of the most mportant technques for generatng poses wth nematc constrants. Researchers have studed IK ntensvely n both computer anmaton and robotcs. here are three dstnct approaches: analytcal numercal and data-drven. Analytcal IK methods use a closed-form nverse of nonlnear nematcs functons. hs approach performs well for some classes of smple structures such as a sngle human lmb. However t lacs suffcent generalty for posng a full-body human character usng a small set of nematc constrants. Numercal algorthms formulate the IK problem as a nonlnear optmzaton problem by mnmzng a cost functon that measures how well a reconstructed pose matches nematc constrants. 4 Numercal IK methods have two advantages. Frst they apply to general artculated bodes such as full-body human characters. Second they can easly ntegrate varous forms of nematc constrants such as pont and orentaton constrants nto an optmzaton. However IK s usually an ll-posed problem when appled to full-body human characters that s many dsparate and unnatural solutons can be consstent wth the nematc constrants. So users must usually specfy a large constrant set to remove the reconstructon ambguty. Our research bulds on the success of data-drven methods. Several researchers have explored varous data-drven IK algorthms to pose full-body human characters5 8 or deformable objects.9 For example Charles Rose and hs colleagues used a weghted combnaton of example poses to constran the IK soluton space. 5 Keth Grochow and hs colleagues appled a global nonlnear dmensonalty reducton technque the scaled gaussan process latent varable model (SGPLVM) to a small set of human-moton data. hey used the constructed probablstc model to compute a most-lely pose from user-defned constrants.6 he Rose and Grochow methods wor well for a small amount of tranng data but don t scale well to the sze and heterogenety of a large tranng database. Recently Xaomao Wu and hs colleagues employed an adaptve clusterng algorthm to select representatve frames from a large moton-capture database to accelerate tranng and posng for SGPLVM.8 Jnxang Cha and Jessca Hodgns used a seres of local statstcal pose models constructed at runtme to reconstruct motons from contnuous lowdmensonal control sgnals obtaned from vdeo cameras.7 However ther method requres the nematc constrants to be contnuous and nown n advance (for example that only hand postons wll be constraned). = arg max q pr (q c) q = arg max q pr (c q) pr (q) pr (c) arg max q pr (c q) pr (q) We based our data-drven model of our algorthm on mxture of factor analyzers (MFAs)0 whch can effcently model pror nformaton embedded n a huge heterogeneous database wth a small number of parameters. Manfred Lau and hs colleagues appled MFAs to prerecorded facal-expresson meshes and used them to constran the soluton space of nteractve facal-expresson modelng. In the man artcle we extend the dea by applyng t to posng a 3D full-body human character. References. J.U. Koren and N.I. Badler echnques for Generatng the Goal-Drected Moton of Artculated Structures IEEE rans. Computer Graphcs and Applcatons vol. no pp M. Grard and A.A. Macejews Computatonal Modelng for the Computer Anmaton of Legged Fgures Proc. Sggraph ACM Press 985 pp J. Zhao and N.I. Badler Inverse Knematcs Postonng Usng Nonlnear Programmng for Hghly Artculated Fgures ACM rans. Graphcs vol. 3 no pp K. Yamane and Y. Naamura Dynamcs Flter Concept and Implementaton of Onlne Moton Generator for Human Fgures IEEE rans. Robotcs and Automaton vol. 9 no pp C. Rose P.-P. Sloan and M. Cohen Artst-Drected InverseKnematcs Usng Radal Bass Functon Interpolaton Computer Graphcs Forum vol. 0 no pp K. Grochow et al. Style-Based Inverse Knematcs ACM rans. Graphcs vol. 3 no pp J. Cha and J. Hodgns Performance Anmaton from LowDmensonal Control Sgnals ACM rans. Graphcs vol. 4 no pp X. Wu L. Reveret and M. ourner Interactve Character Posng from Large Moton Database Proc. ACM Sggraph/ Eurographcs Symp. Computer Anmaton (poster) Eurographcs Assoc. 008 pp.. 9. R.W. Sumner et al. Mesh-Based Inverse Knematcs ACM rans. Graphcs vol. 4 no pp Z. Ghahraman and G.E. Hnton he EM Algorthm for Mxtures of Factor Analyzers tech. report CRG-R-96- Unv. of oronto M. Lau et al. Face Poser: Interactve Facal Modelng Usng Model Prors Proc. 007 ACM Sggraph/Eurographcs Symp. Computer Anmaton (SCA 07) Eurographcs Assoc. 007 pp where pr(c) s a normalzed constant and s dropped off durng optmzaton. In our mplementaton we mnmze the negatve logarthm of the posteror probablty dstrbuton functon pr(q c) yeldng the followng IEEE Computer Graphcs and Applcatons 79

3 Feature Artcle energy-mnmzaton problem: argmnq ln pr( cq)+ ln pr( q) E lelhood E pror where E lelhood s the lelhood term whch measures how well the generated pose q matches the user nput c and E pror s the pror term whch descrbes the pror dstrbuton of human poses. Conceptually the pror term measures the naturalness of the syntheszed poses. An optmal estmate of the syntheszed pose produces a natural human pose that acheves the user-specfed goal. he MAP framewor presents two advantages for human-pose synthess. Frst t lets us formulate the synthess n a contnuous optmzaton framewor. he framewor s flexble; we can ntegrate any form of nematc constrant as long as we can numercally evaluate the correspondng lelhood terms that s how well the generated pose q matches user nput c. Second the framewor naturally combnes the user-constrant term wth the pose pror. In our system we let users dynamcally control the weght of ther constrant terms and thereby balance the trade-off between the two terms. hs s mportant because a novce user mght specfy unnatural constrants n that no natural pose precsely matches them. Modelng Natural Human Pose Prors Consder the D screen poston of a left hand. An nfnte number of unnatural human poses can satsfy the constrants t poses. o remove ths nherent ambguty n computng 3D human poses from a small number of nematc constrants we constran the estmated poses to le n the space of natural poses. he frst step s to model the pose prors from the prerecorded moton-capture database. We begn by representng each pose n the database as a 40-dmensonal vector q R 40 n the jont-angle space; we exclude each pose s root postons and orentatons. o model the human pose prors we apply mxture of factor analyzers (MFAs). 34 he MFAs descrbe the hgh-dmensonal pose space wth a probablstc combnaton of dfferent local regons each modeled by a factor analyzer wth a small number of latent varables. he MFAs provde a probablty densty functon (PDF) pr(q) over an entre pose space. Factor Analyzer Models A factor analyzer s a parametrc statstcal model that represents a hgh-dmensonal real-valued data vector q R D such as a human pose usng a low-dmensonal vector of hdden factors s R d and a multvarate Gaussan random vector u R D. In mathematcal terms we can defne a generatve factor-analyzer model as q = Ls + u where L R D d s a factor-loadng matrx. We assume s s a zero-mean Gaussan dstrbuton N(0 I). We assume u has a multvarate Gaussan dstrbuton N(m y) where m s a mean vector and y s a dagonal covarance matrx. Accordng to ths model the hgh-dmensonal data vector q s also Gaussan dstrbuted: ( ) pr( q)= N q µ LL + ψ. he unnown parameters of factor analyzers are the m L and y values that best model the nput data s covarance structure. he factor varables s model correlatons between each component of q whereas the varables u account for ndependent nose n each component of q. MFAs Conceptually all human poses n the database form a hgh-dmensonal nonlnear manfold n the character confguraton space. A global lnear parametrc model such as a factor analyzer s therefore seldom suffcent to capture the nonlnear structure of natural human poses. A better soluton s to use MFAs to model the nonlnear pose dstrbuton. MFAs probablstcally partton the entre confguraton space q R D nto multple local regons and then model the data dstrbuton n each local regon usng a dfferent factor analyzer. MFAs can be thought of as reduced-dmenson Gaussan mxture models. Mathematcally MFAs assume a mxture of normal dstrbutons for the nput data q: K pr( q)= π N ( q µ LL + ψ ) () = where the constant K s the number of factor analyzers. he scalar p matrx L and vector m are respectvely the mxng proporton factor-loadng matrx and mean vector of the th factor analyzer. he matrx y R D D s a dagonal covarance matrx for ndependent nose u. he MFA parameters thus nclude {p m L y = K} where K p =. Whereas n the factor analyzer the = data mean μ was rrelevant and was subtracted before fttng the covarance structure here we have the freedom to gve each factor analyzer a dfferent mean m. hs lets each factor model the data covarance structure n a dfferent part of the nput space. 80 July/August 0

4 Expectaton-Maxmzaton Algorthm We use the expectaton-maxmzaton (EM) algorthm to estmate the parameters of mxture-of-factor analyzers (MFAs) Q = {p m L y = K}. We frst ntroduce a K-dmensonal bnary vector w = [w w K ] satsfyng w {0 } and S w =. We assgn w = when the th factor analyzer generates a data pont. Wth ths mxture ndcator we can wrte the loglelhood of complete tranng data: L = ln p( q s w Θ) = ln p( q s w Θ )+ ln p( q w). he EM algorthm ams to maxmze the expected L. Because the second term of L s ndependent of Q maxmzng the expected L s equvalent to maxmzng the expectaton of the frst term Q: Q = E ln p( q s w Θ) = E lnn q µ + Ls ψ N ( ) w = E ln ( q Hs ψ) w N = c ln ψ Atr ( H ψ HC )+ + A q ψ q A q ψ HB where c s a constant H = (L m ) s B A C = E( w q ) E( w) = sq = E ( ss q w) E( s q w). E( sq w) I s = ( ) and o maxmze Q we alternate between the E-step and the M-step untl Q converges to a local maxmum. E-step Gven the current parameters Q we need to evaluate the followng expected posteror dstrbutons of the latent varables to compute A B and C : ( ) E( w q) π N q µ LL + ψ E( sq w)= β ( q µ ) E ( ss q w)= I βl + β ( q µ ) ( q µ ) β ( ). Usng a Woodbury matrx dentty where β = L ψ+ LL to precompute ( y + LL ) speeds the evaluaton. hese expected posteror dstrbutons computatonal complexty s O(KND ) O(KNDd) and O(KND ) respectvely for the whole tranng data. M-step We eep A B and C constant and maxmze Q wth respect to H y and p. By settng Q/ H and Q/ y to zero we can update the model parameters as follows: new H = A B AjC j q j = new N dag A q H B ( ) y new q where the dag( ) operator constrans y new to be a dagonal matrx. We compute the new mxture coeffcents p new by emprcal measure: p new. N = A he computatonal complexty of H new y new and p new s O(NDK) O(KND ) and O(N) respectvely. So the overall complexty for each EM teraton s O(max(NKD NK D)). Intalzaton We ntalze the model parameters as follows: ψ π 0 0 = dag ( Σ) = K 0 µ = µ + sσ 0 L = s Σ D d j where S and m are the covarance and mean of the whole tranng data and s s a random varable followng N(0 ). he model-learnng process automatcally fnds the model parameters Q = {p m L y = K} from the tranng data q n n = N where N s the number of poses n the database. As s typcal for MFAs we use an expectaton-maxmzaton (EM) algorthm to learn the model parameters. (he related sdebar descrbes ths algorthm for MFAs and how to ntalze t.) In our experment we set the number of factor analyzers (K) to 70 and the dmenson of the latent space (d) to 5. IEEE Computer Graphcs and Applcatons 8

5 Feature Artcle Specfcally the drect-manpulaton nterfaces support four nds of real-tme pose manpulatons. (a) Fgure. Real-tme manpulaton usng D pont constrants. (a) he user selects a character pont (red) and specfes ts poston n the screen Alternatvely we could use cross-valdaton technques to set approprate values for K and d. he learned MFAs can be used to randomly generate an nfnte number of natural human poses by samplng the mxture dstrbuton n Equaton. However we choose to ncorporate them nto the MAP framewor descrbed earler to generate a natural pose q that satsfes user-defned constrants c. We mnmze the negatve log of pr(q) yeldng ths pror term: K = (b) space (green). (b) he system generates a 3D pose. ( ) Epror ( q)= ln π N q µ LL + ψ. () A smaller E pror means that q s closer to poses n the tranng database and therefore s more natural. Learnng MFAs requres O(max(NKD NK D)) for each EM teraton (see the Expectaton- Maxmzaton Algorthm sdebar) whch s lnearly dependent on the sze of the tranng database N. he pror-term evaluaton (see Equaton ) s also effcent; t requres only O(KD ) to evaluate a new pose wth the precomputed nverse covarance matrces. Intutve 3D Character Posng o let novce users of our system create a desred human pose qucly and easly we developed drectmanpulaton nterfaces and setchng nterfaces. Drect-Manpulaton Interfaces In these nterfaces the system starts wth a default 3D human pose. he user can modfy that pose n real tme by draggng any character ponts adjustng dstances between any two character ponts or specfyng orentatons for any bone segments. All the constrants can be specfed on the D screen space. We thereby avod the need for complex 3D nteractons whch can be cumbersome for a novce user. D or 3D pont constrants. Wth D pont constrants the user can select any 3D pont x R 3 on the character and specfy where the pont should map to on the screen space y R (see Fgure ). he 3D poston of the selected character pont x R 3 s defned n the local reference frame. Users can acheve ths by clcng a D screen pont and lettng the system ray-trace the clced pont to fnd the ntersecton between the correspondng 3D ray and the character surface. Gven a set of D pont constrants y = N p the problem s to fnd a character pose (q) to ensure each of the selected 3D ponts (x ) projects onto the correspondng D screen poston (y ) n the current camera vewpont (see Fgure ). Assumng the D pont constrants are ndependent and each s satsfed up to an addtve zero-mean Gaussan nose wth a standard devaton s p we can defne the lelhood term for D pont constrants: E p = ln pr( yq) = ln N p = exp π ( ) y PFqx ( ; ) N PFqx ( ; p ) y = ( ) σ p σ p where the functon F( ) s a forward-nematcs functon that maps the local coordnates of the 3D character pont x under the current character pose q to a global 3D end-effector poston. Note that we drop the constant ln( p ) n the equaton. he functon P( ) s a projecton functon that maps a 3D pont onto a D screen pont under the current camera vewpont. A good match between the projected D ponts and the user-specfed D ponts results n a low value for ths term. We could also pose a character wth 3D pont constrants z = N 3 p. o defne a 3D pont constrant the user frst specfes the projecton of the 3D target poston n the current screen space and uses t to shoot a ray nto 3D space. he user then changes the camera vewpont and selects a 3D target pont along the 3D ray. We defne the lelhood term for 3D pont constrants as N3 p ( ) E3 p = Fqx ; z = s3p 8 July/August 0

6 (a) (b) (c) (d) Fgure. he dfference between 3D and D dstance constrants. (a) wo selected character ponts. (b) he 3D pose generated by the zero 3D dstance. (c) he same pose n a dfferent vewpont. (d) he generated 3D pose wth the D dstance. where s 3p s the standard devaton of addtve Gaussan nose present n 3D pont constrants. D or 3D dstance constrants. Dstance constrants let the user select any two ponts on the character model (x and x j ) and adjust the dstance between them. Fgures a through c show nteractve character posng wth 3D dstance constrants. Let D j represent 3D target dstance constrants specfed by the user and let N c represent the number of dstance constrants. he lelhood term for the 3D dstance constrants s E 3d ( ( ) ( j) Dj ) = Fqx ; F qx ; s 3d where the summaton s for all pars of j and s 3d s the standard devaton for 3D dstance constrants. he user can also control the screen dstance of two character ponts d j. Let y and y j represent the screen ponts coordnates. he lelhood term s E d ( j dj = y y ) sd ( PFqx ( ( ; ) ) PFqx ( ( ; j) ) dj ) = s d where s d s the standard devaton for D dstance constrants. he D dstance constrant s partcularly useful for posng 3D characters from photographs. Fgure shows the dfference between the 3D and D dstance constrants. Bone orentaton constrants. he user can modfy a 3D pose by selectng a partcular bone segment and specfyng ts orentaton n screen space (see Fgure 3). We assume that orentaton constrants are ndependent and are satsfed up to an addtve Gaussan nose wth a standard devaton s o. We (a) can defne the followng lelhood term for the orentaton constrants: E o N o = ( ( )) PFqv ; a s = o PFqv ( ( ; ) ) a (b) Fgure 3. Orentaton constrants. (a) he user selects two bone segments and specfes the desred orentatons on screen. (b) he resultng 3D pose. where a represents the desred orentaton for the th bone segment and N o s the number of the bone orentaton constrants. he vector v s the 3D orentaton of the selected bone on the local reference frame. he operator represents the cross product of two vectors. Local control: Fxed constrants. Human moton nvolves hghly coordnated movement and the movements between dfferent degrees of freedom aren t ndependent. he pose prors constructed from captured moton data effcently model the correlaton between dfferent degrees of freedom n human poses. If the user poses a human character wth one of the constrants we defned earler such as pont constrants the regons of the IEEE Computer Graphcs and Applcatons 83

7 Feature Artcle (a) (b) (c) Fgure 4. Fxed constrants offer local control over 3D poses. (a) he dstance constrant specfed by the user. (b) he fxed constrant defned on the shoulder. (c) he resultng pose from a new vewpont. Wthout the fxed constrant reducng the 3D dstance between the rght leg and left hand wll change the left shoulder s poston. (a) (b) (c) (d) Fgure 5. Setchng nterfaces. (a) Lmb-setchng constrants (red) and spne-setchng constrants (pn). (b) he system converts each lmb setch nto one D pont constrant and two orentaton constrants and converts the torso setch nto four D pont constrants. (c) he syntheszed pose n the same vewpont. (d) he syntheszed pose from a dfferent vewpont. character model that the user doesn t select mght stll change. For example f the user selects and moves the left foot the hand postons can also change even f no constrants are mposed on the hands. Fxed constrants let the user select any character pont that should reman unchanged. hs constrant must be used together wth other constrants thereby allowng for local control of the character poses that are beng changed. We use the fxed constrants to let some bones mantan ther postons n the optmzaton (see Fgure 4). Let x represent the th fxed pont on the local reference frame. Let q 0 be the orgnal 3D pose of the bone segment. he lelhood term for the fxed constrants s N f ( ) ( ) E f = Fqx ; F q ; x 0 = s f where F(q 0 ; x ) represents the global coordnates of 3D ponts to be fxed N f s the total number of fxed ponts and s f s the standard devaton for the fxed constrants. he smaller s f s the more these ponts wll try to stay the same. Setchng Interfaces In setchng nterfaces users can pc any lmb or the torso and draw a D setch to specfy ts desred locaton on screen. he system automatcally generates a natural-loong 3D human pose that best matches the setch. he current system supports two nds of setch constrants: lmb setchng and spne setchng (see Fgure 5). Lmb setchng. hs method lets the user place a seres of nematc constrants on multple connected bone segments (for example the left leg s bones ) n one step thereby sgnfcantly speedng up character posng. In our system the user can draw a D setch to specfy a desred 3D pose for a partcular lmb (see Fgure 5a). he system automatcally nfers the D locatons of multple jonts located on the selected lmb as well as the D drectons of multple bone segments and uses them along wth pose prors to generate a natural-loong pose. 84 July/August 0

8 he lelhood term for the lmb-setchng constrants drectly measures how well the syntheszed pose matches the drawn setches. However ths evaluaton requres nown correspondences between the setched lmb and the syntheszed lmb. he computer vson lterature has addressed setch recognton n numerous ways but we re solvng a much smpler problem one that benefts from addtonal constrants. Each recorded stroe s already nown to be assocated wth a partcular lmb; furthermore the connectvty of the bone lns s nown n advance. We frst apply a Hough transform to the nput setch to automatcally detect a set of lne segments n that setch. he detected lne segments defne each bone segment s orentaton constrants. he system automatcally estmates D jont locatons by computng the ntersectons of adjacent lnes. hs lets us convert the lmb-setchng constrants nto a set of D pont constrants and orentaton constrants (see Fgure 5b). When the nput setch s a straght lne we opt to use only the D orentaton constrants for character posng because we can t determne D jont locatons. Spne setchng. he character torso loos more le a smooth curve than connectng lne segments because the spne supports t. So the lmb-setchng nterfaces mght not be approprate for spne setchng. o address ths problem the system lets the user draw a smooth curve to ndcate the projecton of the entre spne on the screen space (see the pn setch n Fgure 5a). In ths artcle we represent the character s torso wth three bone segments: root lower bac and upper bac. he system automatcally transforms the setched spne nto four D pont constrants each on one spnal jont (see Fgure 5b). We use a smple scheme to dentfy the D pont constrants for each jont on the torso. We assume that the bone segments length ratos are preserved when our system projects the spne from 3D space to D screen space. hs s true when the spne s parallel to the camera plane. Otherwse the user can rotate the vrtual camera to ensure the spne s approxmately parallel to the camera plane. We can use the nown length of each bone segment on the torso to detect the D jont locatons (see Fgure 5b). Runtme Optmzaton Durng runtme the system optmzes the pose n jont-angle space and fnds a most lely pose that satsfes the user constrants. In our mplementaton the optmzaton also enforces jont-anglelmt constrants. Each MFA factor has a mean character pose. We ntalze the optmzaton wth the best mean pose among all factors. We analytcally derve the Jacoban matrces for each term and perform the nonlnear optmzaton usng the Levenberg- Marquardt algorthm wth the levmar lbrary s boundary constrants. 5 he good ntal guess and analytcally evaluated Jacoban matrces ensure that the optmzaton converges rapdly. he optmzaton typcally taes 50 to 00 ms dependng on the number of user-specfed constrants. Durng runtme our system optmzes the pose n jont-angle space and fnds a most lely pose that satsfes user constrants and jont-angle lmts. System Evaluaton Our system generates a varety of natural poses wth mnmal user nput and performs well compared to numercal and alternatve data-drven IK algorthms. he evaluaton results we descrbe here are best vewed n the companon vdeo to ths artcle whch s avalable at org/portal/web/computngnow/cga/vdeos. ranng Data he CMU mocap onlne database that we used for tranng conssts of.8 mllon prerecorded human poses (about 6.48 hours of anmaton). he typcal human behavors are locomoton (jumpng runnng hoppng and walng) physcal actvtes (basetball boxng dance exercse golf martal arts and swmmng) nteractng wth the envronment (step stool rough terran and playground equpment) and common scenaros (cleanng watng and gesturng). In our experment we set the number of factor analyzers (K) to 70 and the dmenson of the latent space (d) to 5. he learned model too up 00 Kbytes. he total tranng tme was approxmately 5 hours. Results wth Dfferent User Constrants We ve demonstrated our system s performance wth both the drect-manpulaton and setchng nterfaces. he accompanyng vdeo shows how users can manpulate the 3D character pose n real tme wth the four types of drect-manpulaton constrants and how they can pose a character IEEE Computer Graphcs and Applcatons 85

9 Feature Artcle (a) (b) (c) Fgure 6. Posng a character accordng to reference photos downloaded from the Internet. (a) Reference photos. (b) Knematc constrants from the reference photos. (c) he resultng poses n four dfferent vewponts. Our system lets novce users nteractvely pose a 3D human character accordng to any reference photo. (Source of football photo: NFL; used wth permsson.) wth both the lmb-setchng and torso-setchng nterfaces. Novce users can nteractvely pose a 3D human character accordng to any reference photos (for example Internet photos). hs powerful feature contrasts wth prevous data-drven IK systems that are based on a small tranng data set and therefore aren t approprate for ths applcaton. Fgure 6 shows four dstnct poses that our drectmanpulaton and setchng nterfaces generated on the bass of reference photos we downloaded from the Internet. As the accompanyng vdeo shows a novce user can also specfy a desred pose by posng an artculated toy n front of a dgtal camera tang a snapshot of the posed toy and usng the drectmanpulaton nterfaces to transform the toy s pose nto a 3D pose. he toy nterface has proved extremely effcent n helpng novce users create a desred pose. Our system demonstrates a strong generalzaton capablty and can generate poses very dfferent from the examples n the CMU mocap database. For nstance Fgure 7 shows the fve closest database poses to the Dscobolus statue n Fgure 6a. None of the closest examples are smlar to the reconstructed pose n Fgure 6b. Nevertheless wthout the pose prors the qualty of the system s generated poses would depend on ether accurate user specfcaton of a detaled constrant set or a very good ntal pose. When the user s nputs conflct wth the pose prors the system allows a trade-off between satsfyng user constrants and creatng natural-loong poses through adjustment of the constrant term weghts n the objectve functon. he user can adjust the weghts that s standard devatons on the fly thereby provdng an nterestng spectrum of choces. Comparsons wth Other Systems We compared our system to fve alternatves usng two databases: the whole CMU mocap database and a database of poses randomly sampled from the whole CMU database. 86 July/August 0

10 Fgure 7. he fve closest poses n the tranng database for the Dscobolus statue photo n Fgure 6a. he dfferences between these poses and our system s reconstructed pose (see Fgure 6b) demonstrate the system s strong generalzaton capablty. able. Average reconstructon errors for dfferent methods and database szes.* Power Error wth the 5000-pose database Error wth the.8 mllon-pose database Inverse nematcs Prncpal component analyss Probablstc prncpal component analyss Scaled Gaussan process latent varable model.9 n/a Lnear prncpal component analyss Mxture of factor analyzers *We computed the reconstructon errors from the same set of constrants (sx 3D constrants from two shoulders two feet and two hands). We used cross-valdaton to compute the dfferent algorthms reconstructon errors wth the same set of nematc constrants. For each testng pose we generated sx 3D pont constrants (both shoulders feet and hands) and used them to reconstruct a correspondng 3D pose. he reconstructon error measures the average L dstance between the ground-truth poses and the reconstructed poses n poston space that s the pont clouds. able summarzes the numercal reconstructon errors for the two testng databases. IK uses numercal optmzaton to compute ts solutons. Prncpal component analyss (PCA) performs the optmzaton n the PCA subspace wthout any pror term. Probablstc PCA (PPCA) conducts the optmzaton n the PCA subspace wth a Gaussan pror term assumng that a multvarate Gaussan dstrbuton represents the database samples. Scaled Gaussan process latent varable model (SGPLVM) learns a global nonlnear embeddng. Lnear PCA (LPCA) fnds K database poses that best match the user constrants and performs the optmzaton n the PCA subspace computed by the K nearest examples. 6 All these methods start wth the same ntal pose and optmze the pose usng the Levenberg-Marquardt algorthm. Our method wth prors constructed from the large database produces the smallest reconstructon error. IK and PCA have the largest errors because the number of nematc constrants sn t suffcent to determne a full-body pose. PPCA ntroduces a Gaussan pror to reduce the reconstructon ambguty thereby producng more accurate results than PCA. SGPLVM whch s based on 6D latent space and an actve set sze of 00 generates better results than IK PPCA and PCA for the small database. Evaluatng ts performance for the large database s computatonally too expensve. LPCA produces low errors on both databases. However t s computatonally neffectve because t must search the whole database to fnd the K closest examples. For all these methods the prors learned from the larger database result n lower reconstructon errors; ths demonstrates the necessty of modelng prors from a huge heterogeneous moton-capture database. We also compared the dfferent algorthms effectveness for posng a character from reference photos. Fgure 8 shows the 3D poses reconstructed from the football player photo n Fgure 6a. he Importance of Pose Prors We evaluated how dfferent pose prors nfluence the fnal moton under the same set of user-defned constrants. We frst computed a set of constrants from a random runnng pose that s not n the IEEE Computer Graphcs and Applcatons 87

11 Feature Artcle (a) (b) Fgure 8. Alternatve methods of modelng pose prors on the football player example n Fgure 6a. (a) Results from the orgnal vewpont. (b) Results from a dfferent vewpont. he pose generated usng MFAs not only matches the nput mage but also loos natural. database. We then compared the results usng prors constructed from a runnng database and from the whole database. he accompanyng vdeo shows that we can generate a good runnng pose wth both prors. We also computed a set of constrants from a random sttng pose that s not n the database. As we expected the system faled to generate a good sttng pose when the pose pror s learned from runnng. Our system lets us model a wde range of poses that can t be generalzed from a small amount of tranng data. We beleve ths capablty wll become more and more mportant for future applcatons as the avalablty and popularty of moton-capture data ncrease. References. C. Rose P.-P. Sloan and M. Cohen Artst-Drected Inverse-Knematcs Usng Radal Bass Functon Interpolaton Computer Graphcs Forum vol. 0 no pp K. Grochow et al. Style-Based Inverse Knematcs ACM rans. Graphcs vol. 3 no pp Z. Ghahraman and G.E. Hnton he EM Algorthm for Mxtures of Factor Analyzers tech. report CRG- R-96- Unv. of oronto M. Lau et al. Face Poser: Interactve Facal Modelng Usng Model Prors Proc. 007 ACM Sggraph/ Eurographcs Symp. Computer Anmaton (SCA 07) Eurographcs Assoc. 007 pp M.I.A. Louras levmar: Levenberg-Marquardt Nonlnear Least Squares Algorthms n C/C++ 009; 6. J. Cha and J. Hodgns Performance Anmaton from Low-Dmensonal Control Sgnals ACM rans. Graphcs vol. 4 no pp Xaoln K. We s a PhD student and research assstant n computer scence at exas A&M Unversty. Hs research nterests nclude data-drven character anmaton vdeobased moton capture nteractve character-moton authorng physcs-based anmaton moton control and computer vson. We has a bachelor n computer scence from Xdan Unversty. Contact hm at xwe@cse.tamu.edu. Jnxang Cha s an assstant professor n exas A&M Unversty s Department of Computer Scence and Engneerng. Hs man research nterests are character anmaton datadrven approaches to graphcs and vson nteracton technques for 3D graphcs vson for graphcs and anmaton and mage-based renderng and modelng. Cha has a PhD n robotcs from Carnege Mellon Unversty. Contact hm at jcha@cse.tamu.edu. Selected CS artcles and columns are also avalable for free at 88 July/August 0