3D model-based human modeling and tracking

Transcription

1 3D model-based human modeling and tracking André Gagalowicz Projet MIRAGES INRIA - Rocquencourt - Domaine de Voluceau Le Chesnay Cedex Andre.Gagalowicz@inria.fr

2 FORMER APPROACH 2

3 Golf-Stream Research Project Started: June 2002 Duration: 33 months. Grant: RIAM: Centre National du Cinéma, Ministère de l industrie. Partners: INRIA (Institut National de Recherche Informatique et Automatique) FFG (Fédération Française de Golf) PGA-France (Professional Golfers Association) SYMAH VISION (Lagardere Group): coordinator 3

4 Existing Systems for Swing Analysis Motion Capture (Vicon, Motion Analysis): Motions analysis from optical capture. Reflecting Markers are automatically segmented and tracked using stroboscopic lights. Limitations: - Weakness of marker tracking, - Sliding of markers on the body - Indoor environment only - ->Not in Real Conditions (even Psychologically) 4

5 State of the Art of 3D Human Motion Analysis Image Analysis (mono or multi cameras) Gavrila & Al. I. Mikic, M. Trivedi, R. Jain I. Kakadiaris & Al. P.Fua &Al. 5

6 Overview of the human body tracking (1) Golf-Stream Project 6

7 Overview of the human body tracking(2) 3D Tracking 7

8 Filming Conditions: - 7 Cameras are located on the fairway at a starting point. - - Shooting of the Trophée Lancôme tournement of the 4 first national champions, in almost real conditions. - Image rate: 50Hz (analysis of field) - Genlocked Cameras - Shutter: 1/500s (no blur on some body part) - Simultaneous meta-data recorded (3D ball distances, sportive comments, Ball trajectories, Impact sounds, Anthropometric measurements, ) 8

9 Filming Conditions: 9

10 Camera calibration: A 3D tool whose 3D geometry is known is used to make 2D/3D correspondances and to retreive the 3D camera parameters (extrinsics & Extrinsics). A simple pinhole camera model is used. 10

11 Design of a 3D puppet (1) Generic Mesh - Use of a standard human 3D mesh, corresponding to standard human morphology. 11

12 Design of a 3D puppet (2) Anthropometric adjustment - Stastistical 3D human body adjustment according to the Ergoman study (L.A.A) - 22 measures are necessary to preadjust our generic puppet to a specific human body. 12

13 Design of a 3D puppet (3) Image Refinement - Several static positions are used to refine the pre-adjusted puppet to the champion to track. 13

14 Design of a 3D puppet (4) 3D skeleton - Anatomical skeleton design: - - Better approximation of the real human anatomy (biomechanic constraints) - - Easier Skinning - - Better pivot location 14

15 Design of a 3D puppet (5) Animating a 3D puppet - Skin mesh deformation is related to bones motions, using a skinning function. - Parametric puppet. 15

16 Tracking (1) Pre-positionning a 3D puppet on the 1 st image - Interactive ajustment of the 3D puppet on the 1 st image on each view. The implicit biomechanical contraints of the puppet helps the user to find a realistic posture. 16

17 - 3D mesh vertices are projected in each camera plane on the first frame. - These projections represent the texture coordinates in every views. - According to a criterion of good visibility of a polygon in a camera view (occlusion and dot product value between its normal and the camera optical axis), several textures might be stored for the same polygon -> PROBLEM: colorimetry consistency among time and space for the textures. Tracking (2) Texture Learning 17

18 Tracking (3) Matching Process between synthetic puppet and Images - To compute a matching error, the algorithm render as many synthetic images of the textured puppet as the number of available cameras. - The matching error is given by: 18

19 Tracking (4) Active DOF and Hierarchical Matching - - Tracking a human body using the synthetic puppet consists in retrieving the DOF values of the skeleton along time, using an interactive process that compare appearance of that puppet for each trial with the real images. - - Since the matching error cannot be evaluated analytically, Simulated Annealing is used to perform the minimization. - The search of the puppet posture at Image (t+1) is made using the known puppet pose at Image (t) as an input. 19

20 Tracking (5) Active DOF and Hierarchical Matching - Skeleton Hierarchical search (6 DOF for Pelvis: global position, 3 DOF for Spine, 2 for each Upper-Leg, ) Body skin appears only when the corresponding bone(s) have been or are currently tracked and only that part is matched with the real images. - Limitation of the parameter space by constraining the optimization (When the Simulated annealing is trying a not realistic position, the matching is not computed and a high value is return to avoid the optimiser to not try this parameter value again..) 20

21 Some mesurements obtained from swing tracking. 21

22 Future Works - Computing Tracking from the data set. - Make more tasks automatic: 3D puppet adjustments, prepositioning - Using these results to create a swing motion engine able to predict a new swing motion tracking. - Being able to track golfers wearing normal cloth (Next filming: Open de France tournament, June 2004). - Speed up the computation. - Adapt this method to other outdoor sports (athletics) and complex motions (dance). 22

23 Conclusion - We are currently developing a 3D tracking system using video cameras able to make outdoor motion capture without using markers. - A standard 3D human body and anthropometric measurements give the very first result of the morphology adaptation. - Further body adjustments are made using images. - After positioning the puppet on the first frame, the texture is learnt and a tracker will retrieve the position of the puppet in the rest of the sequence. 23

24 Modeling 3D humans from uncalibrated wide baseline views NEW APPROACH

25 Context of this research (the same!) A system for 3D-tracking of a golf swing We need a precise and truthful specific 3D model to work with for tracking Indeed, the quality of tracking results relies on the quality of the 3D model 25

26 Block diagram 26

27 The images used 27

28 The 3D generic model Made of about vertices facets 28

29 Block diagram 29

30 Calibration Made using POSIT and 3D characteristic points and their correspondents on images 30

31 Reconstruction From calibration, reconstruct the 3D characteristic points 31

32 Block diagram 32

33 RBF interpolation 33

34 34 RBF interpolation Apply: With: Indeed, we have f(p i )=r i For smoother results, we take s(r)=r Solved by linear algebra ) ( ) ( 1 å = - * = n i i p i p A p f s ú ú ú ú û ù ê ê ê ê ë é = ú ú ú ú û ù ê ê ê ê ë é ú ú ú ú û ù ê ê ê ê ë é n n n n n n r r r A A A p p p p p p p p p p p p M M M O M M O ) ( ) ( ) ( ) (... ) ( ) ( s s s s s s

35 Results 35

36 Block diagram 36

37 3D silhouette extraction From contour edges, we compute silhouette edges using the following algorithm: 37

38 Problem of 2D edge intersections 38

39 Problem of 2D edge intersections 39

40 Curve matching 40

41 3D deformation From 2D to 3D Apply RBF to interpolate 41

42 Block diagram 42 Customized 3D Human Model

43 Final results => Average reprojection error of less than 1 pixel. 43

44 Existing problems 44

45 Model Adjustment ( Surface Smoothing) P( x, y, z) Î SpecificModel, r is a threshold Q ( x, y, z), i = 1,2,..., N is the neighbor vertices of i 1 P' ( x, y, z) = N N å i= 1 if ( dist( p, p' ) > r) p = p' Q ( x, y, z) i Q 1 P P P Q2 Q 3 r = average( dist( p i, pi 45 '))

46 Specific Model Adjustment (Surface Smoothing) r = average ( dist ( p i, p i ' )) Before smooth 46 After smooth

47 Model Adjustment (Slice Replacement) Cut every part of body into slices Precondition: for all human models, the shape of corresponding slices are similar Generic Model Specific Model 47

48 Model Adjustment (Slice Replacement) Match every pair of corresponding slices Overlap every corresponding slice Make the comparison Generic Model Specific Model 48

49 Model Adjustment (Slice Replacement) All the distorted slices are marked in blue Generic Model Specific Model 49

50 Specific Model Adjustment (Slice Replacement) Scaling Replacement Before replacement 50 After replacement

51 Specific Model Adjustment (Slice Replacement) Before replacement After replacement 51

52 Model Adjustment (Detail Replacement) Precondition: the surface shapes between generic and specific model are similar Try to locate the distorted patches by compare the normal of neighbor patches P ÎGenericModel, P' Î SpecificModel V i ( i = 1,2,..., N) is the normal of neighbor patch of P V ' i ( i = 1,2,..., N) is the normal of neighbor patch of P' suspicious( P') = ï ì ' 1 MAX ( f( V i, V )) í > i ïî 0 otherwise r 52

53 Model Adjustment (Detail Replacement) Get the surrounding box Scaling replacement Iterative process 53

54 Model Adjustment (Detail Replacement) Before replacement After replacement 54

55 Model Adjustment (Detail Replacement) The specific model (red one) looks more handsome 55

56 Reconstruction Result 56

57 Texture Mapping Parts of modeling, important for tracking Generate texture on the reconstructed 3D model surface Single VS multiple cameras Viewpoint-Independent Method 57

58 Texture Mapping for Multiple Cameras Viewpoint-Independent Algorithm 1) For each patch p i, do the following processing. 2) Compute the locally normal vector V lmn of patch p i. 3) For each camera c j, compute viewline vector V cj directing toward the centroid Of p i. 4) Select such camera c* that the angle between V lmn and V cj becomes minimum. 5) Extract the texture of p i from the image captured by camera c*. 58

59 Mapping with One Camera 59

60 Mapping with Two Cameras 60

61 Mapping with Six Cameras 61

62 Vertex Blending y = deformed vector position of vertex b = number of bones w n = scalar weight of vertex to bone n x n = original vector position of vertex relative to bone n M n = transform matrix of bone n v y b 1 - = å n= 0 ( w n * v x n M n ) 62

63 63

64 Texture mapping result 64

65 Skeleton Estimation Use features & limb deformation vectors. RBF interpolation. Generic skeleton à Estimated subject s skeleton 65

66 Generic skeleton reconstructed one (face) 66

67 Generic skeleton reconstructed one (side view) 67

68 Results (continued) 68

69 More Results 69

70 Automatic pre-positioning 70

71 3D Motion Tracking 71

72 Human Motion Capture Process Model-based tracking using analysis-by-synthesis. Do NOT rely image segmentation, which is very ill-posed 72

73 3D Model Illustration : Kinematics chain and colour labelling of the human arm parts 73

74 Pre-positioning of arms - before tracking Pre-positioning using 3D software e.g. 3DS Max to do manual adjustment 74

75 Motion Tracking Algorithm 75

76 Synthesize to Search for Correct Posture Numerical minimization as kernel Kinematics of the joints drive the external skin which in turn produce the visual appearance i.e. the posture that we see. Match the synthesize image with the real image. Matching error is feedback & use for synthesize the next search candidate for searching iterative. 76

77 Numerical Optimization Gradient-based e.g. Newton, Levenberg- Marquardt, steepest-decent: less iterations, but converge to local minima. Not applicable here!!) Simulated annealing global minimization, but take more iterations. Particle filter. We chose simulated annealing with simplex search. 77

78 Error Function : Image differences 78

79 Generating the human pose for matching 79

80 Minimize Hierarchically Minimize hierarchically, on different body parts. Rigid body part deformation. Deal with occlusion by minimizing from most visible part first. 80

81 Implementation In C++. Running on Intel Pentium 4 and NVidia 6600 (AGPx8 bus). Maximum of 2 synchronized off-the-shelf cameras. Hardware acceleration using GPU, with the aid of 3D graphics engine Wildmagic. Hardware acceleration needed for 3D graphical rendering!! 81

82 Results See videos Arms tracking: Simple Tracking Cluttered Occlusion Outdoor One frame takes about 8-10 second to computing the tracking result offline. Bottleneck in the transferring of data from GPU to CPU 82

83 SIMPLE TRACKING 83

84 CLUTTERED ENVIRONMENT 84

85 OCCLUSION 85

86 OUTDOOR SCENE 86

87 Results - Comparing accuracy of Automatic Tracking vs Manually Clicked TrackedResult ClickedResult Angle of rotation (degrees) Time Sample (seconds) 87

88 Conclusions Construct 3D human model (with skeleton) from wide base-line images. Camera calibration and 3D reconstruction at the same time from feature points. Refine the feature reconstruction by matching of silhouettes. Model-based tracking of human arms to obtain kinematics of the movement. 88

89 Future Work Extensive validation. Extension to full-body tracking. Take deformable skin into consideration i.e. non-rigid body parts such as twisting of forearm. Automatic pre-positioning of posture. Multiple subject tracking. Computational efficiencies. 89

90 ? Any?? Questions? 90