Robust Human Body Shape and Pose Tracking

Transcription

1 Robust Human Body Shape and Pose Tracking Chun-Hao Huang 1 Edmond Boyer 2 Slobodan Ilic 1 1 Technische Universität München 2 INRIA Grenoble Rhône-Alpes

2 Marker-based motion capture (mocap.) Adventages: precision, reliability little data ( couple of kb/frame for 50 cameras, 5 people) real-time processing, visualization & retargeting. Disadvantages: Attaching, removing, re-attaching markers is tedious. Markers can interfere with the movement. Giant Studios (L.A. Noire set ) Markers prevent the simultaneous acquisition of shape and motion. 2

3 Marker-less mocap. Multiple camera setup is usually required. Accquisition of both motion and shape. 3D dome (CMU) Grimage ( INRIA ) 3

4 Motivation Methods that assume a skeleton usually produce skinning artifacts, require 2 nd stage shape refinement. Degree of freedom (DoF): N J 6 (< 10 2 ) [Vlasic et al. ToG`08] Skeleton-based Purely-surface-based methods handle non-rigid surface deformation better, but do not provide the pose. DoF: N P 6 ( 10 3 ) [Cagniart et al. ECCV`10] Surface-based 4

5 Contribution bone differential coordinate probablistic surface deformation [Straka et al. ECCV`12] [Cagniart et al. ECCV`10] A learning-based outlier rejection scheme. 5

6 Preprocessing step: model Decompose the reference mesh into patches [Cagniart et al. CVPR`10] Shape parameter N, P k k k 1 Θ R c M 0 Pose parameter J N J x j j 1 Rig the skeleton inside the mesh. Each vertex belongs to one body part. [Baran et al. TOG`07] 6

7 Preprocessing step: input data For each time stamp t, visual hull is reconstruced from silhouettes, which serves as our observations 7

8 Pipeline input data t partitioned t filtered t SVM classification filtering M t 1 model Minimizing E(Θ, E(Θ) J) M t 8

9 Pipeline input data t M t 1 model Minimizing E(Θ, E(Θ) J) [Cagniart et al. ECCV`10] M t 9

10 SVM-based body part classification input data partitioned t t SVM classification A multi-class linear SVM is trained on M t 1 t and tested on M t 1 model Feature: 3D coordinate of vertices. Class label: rigid body part label. Good compromise between accuracy and training time. 10

11 Filtering point cloud partitioned t filtered t Bone T b : patches on the bone often move rigidly together. filtering sub-sample the observations. Joint T g : patches on the joint have nonrigid deformation. keep all the observations. Outlier T 0 : abandon all the observations. 11

12 Benefit of SVM classification outlier removal 1 wo outlier removal with outlier removal 0 12

13 Energy minimization filtered t M t 1 model Minimizing E(Θ, J) M t 13

14 Energy function M t 1 E Θ, J filtered t Θ, J Θ Θ Θ, J E E E E data rigid bone E data (Θ) : how well the surface explains the observations. E rigid (Θ) : smooth the motion of neighboring patches. E bone (Θ, J) : keep the relationship between the mesh and the skeleton. 14

15 Data term E data (Θ): how well the surface explains the observations. A probablisitic Iterative Closest Point (ICP) approach. Each observation has a soft assignment to every patch, updated in each iteration. Let observation i correspond to vertex v i k in P k with a soft assigment w i k. E data NT NP1 2 k wi i k vi i1 k1 Θ y x 15

16 A probablistic point of view [Cagniart et al. ECCV`10] Can be interpreted as EM algorithm. The likelihood: Gaussian mixture model E-step: update the soft assignment. P( z k y, Θ) k i i i i N 1 l1 M-step: minimize sum of negative log likelihood (energy). E P P( y z k, Θ) data P( y z l, Θ) i N P i 1 P( y Θ) P( y z k, Θ) i k i i k 1 NT NP1 2 k wi i k vi i1 k1 Θ y x ln( ) 16

17 Rigidity energy [Cagniart et al. CVPR`10] E rigid (Θ) : smooth the motion of neighboring patches P k k x v d l x v P l E rigid N P Θ x x k 1 PN vp P l k k l k v d l v 2 For each patch, the real location and the predicted location should be consistent. Θ is implicitly encoded in x v k and x v l 17

18 Bone-binding energy β coordinate : A relative displacement from patch to bone γ > 1 γ k 0.5 β k c k γ 0 X j(k) Δ k β k = Δ k c k X j_child(k) Δ k = γ k X j(k) + 1 γ k X j_child(k) γ k is computed such that β k is perpendicular to the bone. 18

19 Bone-binding energy E bone (Θ, J) : keep β consistent after transformation. bone N P, E Θ J w β T Θ β k 1 0 k k k k k 2 T k Θ k Δ k 0 N P k 1 w 0 k k k k k Δ T Θ Δ d 2 Δ k d For each patch, Δ k predicted from the patch and Δ k from the bone should be consistent. w k is weighed according to γ k 19

20 Energy function E data (Θ) : how well the surface explains the observations. E E rigid (Θ) : smooth the motion of neighboring patches. E data rigid N N P T N P Θ x x k 1 PN vp P l k k l E bone (Θ, J) : keep the relationship between mesh and joints. 1 Θ w y x i1 k1 k k i i v k v 2 l v 2 regularization terms or deformation prior bone N P, E Θ J w Δ T Θ Δ k 1 0 k k k k k 2 20

21 Minimizing the energy Θ, J Θ Θ Θ, J E E E E d data r rigid b bone λ d =10, λ r =1, and λ b =1 Each term is quadratic in terms of variables. Standard Gauss-Newton optimization is thus feasible. 3-4s per frame (including SVM training time). 21

22 Quantitative results Pose: 70.86mm error in Walking sequence from HumanEvaII benchmark. (error < 80mm typically corresponds to a correct pose [Sigal et al. IJCV`12]). Shape: reprojection error (%) Sequence our surface-based [4] inverse kinematic Handstand 1 [1] Wheel [1] Skirt [1] Dance [1] Crane [2] Handstand 2 [2] Bouncing [2] Free [3] Gall et al. CVPR`09 2. Vlasic et al. ToG`08 3. Starck et al. CGA`07 4.Cagniart et al. ECCV`10 22

23 Qualitative results [Starck et al. 2007] [Vlasic et al. 2008] [Gall et al. 2009] 23

24 Conclusion and future work A method that jointly recovers the pose and the shape of human body has been proposed. We introduce a novel SVM-based classification scheme that filters target point clouds and thus helps better correspondence search. Future directions include alleviating the requirement of background substraction, and exploiting more photometric information. More experiment results in the poster session. 24

25 Thank you! Questions? 25