Combined Shape Analysis of Human Poses and Motion Units for Action Segmentation and Recognition

Transcription

1 Combined Shape Analysis of Human Poses and Motion Units for Action Segmentation and Recognition Maxime Devanne 1,2, Hazem Wannous 1, Stefano Berretti 2, Pietro Pala 2, Mohamed Daoudi 1, and Alberto Del Bimbo 2 1 University Lille 1 /Télécom Lille / CRIStAL, FRANCE 2 University of Florence MICC, ITALY UHA3DS 15, May , Ljubljana, Slovenia

2 Motivation Human behavior analysis challenging task Complexity of human motions Variability of gesture combinations Local "temporal" analysis of a sequence More accurate understanding of what the human is doing Instead of analyzing the whole sequence 2

3 Goal Action segmentation Segmentation of a long sequence into different actions Sequence Walk Jump Run Action recognition Recognize the action performed by a subject in a sequence Especially deal with the failure case of gesture s repetition Action Hand waving 3

4 Cost effective depth sensor Release of cost effective depth cameras Microsoft Kinect, Asus Xtion Pro Live, Depth image in addition to the color image Skeleton estimated from depth image* Depth image Skeleton * Shotton et al. (CVPR 2011) 4

5 Skeletal representation How to represent a 3D human skeleton for action segmentation and recognition? 5

6 Skeletal representation Characterization of a human pose? Location of different body parts relative to each other (shape of pose) Human skeleton representation? Local evolution describing the relative position between various body parts. Representing a 3D human skeleton for action segmentation? Using geometric shape of a set of 3D points A set of 3D points as a curve Analyzing the shape of a curve 6

7 Shape Analysis of curves in R n Curve Representation: Square-Root Velocity Function (SRVF) * SRV Curve in R 3 F Such function q captures the shape of a curve Each function q is an element of the Riemannian manifold called Shape Space Shape comparison between curves q 1, q 2 Geodesic distance between q 1 and q 1 on the shape space S * Joshi et al. (CVPR 2007) 7

8 Pose-based Approach Human pose as a 3D curve instead of a 3D skeleton A set of 3D points (joints) connected by rods (bones) Spatial configuration Joints of each limb are connected together Representing the shape of the body Shape analysis of poses Human pose Geodesic distance between two poses represented on the Shape space 3D Curve Shape space ds q1 q2* Curve 1 Curve 2 Shape space 8

9 Pose-based Approach Motion Segmentation Sliding window technique Within each temporal window Mean pose computation: Riemannian Center of Mass Standard deviation of all poses Karcher mean computing Window p m 9

10 Pose-based Approach Motion Segmentation 10

11 From pose to segment (trajectory) Each skeletal human sequence is segmented into Motion Units What interest? How to represent and exploit MUs? 11

12 Motion Unit representation What characterizes a human motion? Pose changes in the time (segment-based) Motion representation? Temporal evolution describing the relative change between poses. Represent a motion for human action recognition? Using geometric shape of a set of skeletal postures Trajectory : set of skeletal poses Analyzing the shape of a trajectory 12

13 Action recognition by shape analysis on R n * * Devanne et al. (Trans. On Cyb. 2014)

14 Action recognition by shape analysis on Rn Limitations 14

15 Segment-based Approach Detection of periodic Motion Units Sequence : successive MUs Shape Analysis of trajectories representing MUs Distance between trajectories Periodic action segmentation Walking is a succession of «left step» and «right step» Grouping periodic MUs in same action cluster Different lengths 15

16 Experimental Results Action Segmentation 14 sequences from the CMU dataset Mocap Data Long sequences where the subject performs successively different actions Provided ground truth 16

17 Experimental Results Action Segmentation 14 sequences performed by the subject #86 Comparison with the state-of-the-art s method Hierarchical Aligned Cluster Analysis (HACA) [Zhou et al. Trans. On PAMI 14] Ground truth HACA Our Segmentation Result (4th sequence) Segmentation accuracies for 14 sequences (subject #86 ) 17

18 Segment-based Approach Action Recognition Repetition of gestures affect the recognition task Action «hammer» with? hammer strokes Removing the repeated MUs Evaluate how the repetition s removal affect the recognition accuracy Comparing with our previous work * Comparing with a method from the sate of the art * Devanne et al. (Trans. On Cyb. 2014) 18

19 Experimental Results MSR Action 3D dataset [Li et al. HCBA 2010] 20 actions performed by 10 subjects 2-3 times Gaming action without any object Cross-Subject protocol high arm wave side kick jogging Method Accuracy (%) Actionlet [Wang et al. CVPR 12] 88,2 Moving Pose [Zanfir et al. ICCV 13] 91,7 ScTPM [4] [Luo et al. ICCV 13] 93,8 Our previous [Devanne et al. Trans. On Cyb. 2014] 92,1 Our 94,3 19

20 Experimental Results MSRC-12 dataset * 12 iconic and metaphoric gestures 594 sequences of 12 gestures 30 different persons / perform each action several times Each gesture is repeated several times (from 2 to 15) Image from: * Fothergill et al. ACM CHI 2012] 20

21 Experimental Results Results: MSRC-12 dataset Only 296 sequences of 6 gestures classes Comparison with state-of-the-art s method * 5-fold leave-person-out-cross-validation (24 training & 6 test) Class DFM [*] Our previous [**] Our Duck 96, Goggles 88,0 82,0 90,0 Shoot 85,7 73,5 81,6 Throw 90,0 88,0 90,0 Change weapon 87,5 89,6 89,6 Kick 98,0 98,0 98,0 Mean 90,9 88,5 91,5 ** Devanne et al. Trans. On Cyb. 2014] *Lehrmann et al. (CVPR 2014) 21

22 Summary Segmentation of skeletal sequence into Motion Units (MUs) Shape analysis on a Riemannian manifold 3D human pose ND trajectory MUs Competitive results for the task of gesture segmentation Improvement of our previous approach for the task of action recognition Motion segmentation for: Simultaneous segmentation and action recognition (online recognition) Activity recognition Complexity of human motion is increase 22

23 Thank You Questions 23

24 References [1] A. Fod, M. J. Mataric, and O. C. Jenkins. Automated derivation of primitives for movement classification. Autonomous Robots, 2(1):39 54, Jan [2] F. Zhou, F. De la Torre, and J. K. Hodgins. Hierarchical aligned cluster analysis for temporal clustering of human motion. IEEE Trans. on Pattern Analysis and Machine Intelligence, 35(3): , Mar [3] M. Zanfir, M. Leordeanu, and C. Sminchisescu. The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection. In Proc. IEEE Int. Conf. on Computer Vision (ICCV), pages IEEE, [4] J. Luo, W. Wang, and H. Qi. Group sparsity and geometry constrained dictionary learning for action recognition from depth maps. In IEEE Int. Conf. on Computer Vision (ICCV), pages , [5] J. Wang, Z. Liu, Y. Wu, and J. Yuan, Mining actionlet ensemble for action recognition with depth cameras, CVPR [6] M. Devanne, H. Wannous, S. Berretti, P. Pala, M. Daoudi, and A. Del Bimbo. 3D human action recognition by shape analysis of motion trajectories on riemannian manifold. IEEE Trans. On Cybernetics, [7] A. Lehrmann, P. Gehler, and S. Nowozin. Efficient nonlinear markov models for human motion. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages , Columbus, OH, USA, June [8] S. H. Joshi, E. Klassen, A. Srivastava, and I. Jermyn, A novel representation for riemannian analysis of elastic curves in Rn, CVPR

25 Square Root Velocity Function Distance = Great cercle arc Preshape space Shape space 06/10/

26 Pose-based Approach Motion Segmentation 26

27 Proposed Approach Shape analysis of human pose in R 3 Local changes detection Sequence decomposition into motion units Shape Analysis of spatio-temporal motion units (Mus) in R 60 Detection of similar cycles of MUs Grouped for action segmentation Removed for action recognition 27

28 Skeletal representation Characterization of a human pose? how different body parts are located relative to each other (pose-based) Representation of It should be such that its local evolution directly describes the relative position between various body parts. How to represent a 3D human skeleton for action segmentation? Represent a skeleton using geometric shape of a set of 3D points A set of 3D points can be viewed as a curve Analyzing the shape of a curve 28

29 Segment-based Approach Action Recognition Repetition of gestures affect the recognition task Action «hammer» with? hammer strokes Removing the repeated MUs Keep continuity between the two extreme poses Geodesic path Shape Space 29

30 HACA Our 30