Visual Action Recognition

Transcription

1 Visual Action Recognition Ying Wu Electrical Engineering and Computer Science Northwestern University, Evanston, IL / 57

2 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 2 / 57

3 What is an Action? Action: Atomic motion(s) that can be unambiguously distinguished (e.g. sitting down, running). 3 / 57

4 What is an Action? Action: Atomic motion(s) that can be unambiguously distinguished (e.g. sitting down, running). An activity is composed of several actions performed in succession (e.g. dining, meeting a person). 3 / 57

5 What is an Action? Action: Atomic motion(s) that can be unambiguously distinguished (e.g. sitting down, running). An activity is composed of several actions performed in succession (e.g. dining, meeting a person). Event is a combination of activities (e.g. football match, traffic accident). 3 / 57

6 What is Action Recognition? What is Recognition? The recognition of action is to match the observation (e.g. videos) with previously defined patterns and then assign it a label, i.e. action type. Input: an action video; Output: an action label; 4 / 57

7 What is Action Recognition? What is Recognition? Verification: Is the walking man Michael? The recognition of action is to match the observation (e.g. videos) with previously defined patterns and then assign it a label, i.e. action type. Input: an action video; Output: an action label; 4 / 57

8 What is Action Recognition? What is Recognition? Verification: Is the walking man Michael? Identification: Who is the walking man? The recognition of action is to match the observation (e.g. videos) with previously defined patterns and then assign it a label, i.e. action type. Input: an action video; Output: an action label; 4 / 57

9 What is Action Recognition? What is Recognition? Verification: Is the walking man Michael? Identification: Who is the walking man? Recognition: What is the man doing? The recognition of action is to match the observation (e.g. videos) with previously defined patterns and then assign it a label, i.e. action type. Input: an action video; Output: an action label; 4 / 57

10 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; 5 / 57

11 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: 5 / 57

12 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance 5 / 57

13 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance crowd behavior analysis 5 / 57

14 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance crowd behavior analysis human-machine interfaces 5 / 57

15 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance crowd behavior analysis human-machine interfaces sports video analysis 5 / 57

16 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance crowd behavior analysis human-machine interfaces sports video analysis video retrieval 5 / 57

17 Why Need Action Recognition? Expensive human effort to handle rapidly increasing amount of video records; Large number of potential applications: visual surveillance crowd behavior analysis human-machine interfaces sports video analysis video retrieval etc. 5 / 57

18 Main Challenges in Action Recognition Different scales 6 / 57

19 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. 6 / 57

20 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion 6 / 57

21 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter 6 / 57

22 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter Other objects/humans are present in the video frame. 6 / 57

23 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter Other objects/humans are present in the video frame. Partial Occlusions 6 / 57

24 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter Other objects/humans are present in the video frame. Partial Occlusions Human/Action variation (large intra-class variation) 6 / 57

25 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter Other objects/humans are present in the video frame. Partial Occlusions Human/Action variation (large intra-class variation) Walking movements can differ in speed and stride length. 6 / 57

26 Main Challenges in Action Recognition Different scales People may appear at different scales in different videos, yet perform the same action. View changes and camera motion Background clutter Other objects/humans are present in the video frame. Partial Occlusions Human/Action variation (large intra-class variation) Walking movements can differ in speed and stride length. Etc. 6 / 57

27 Action Recognition Methods Handling Space-Time Appearances for Actions Focus on extracting better appearance representation from action video; 7 / 57

28 Action Recognition Methods Handling Space-Time Appearances for Actions Focus on extracting better appearance representation from action video; hand-crafted features: HOG [7], HOF [4], MBH [18] or combinations [18]; 7 / 57

29 Action Recognition Methods Handling Space-Time Appearances for Actions Focus on extracting better appearance representation from action video; hand-crafted features: HOG [7], HOF [4], MBH [18] or combinations [18]; learned features: deep neural network [21, 5, 16] 7 / 57

30 Action Recognition Methods Handling Dynamics for Actions Focus on modeling the dynamics and motions in action video; 8 / 57

31 Action Recognition Methods Handling Dynamics for Actions Focus on modeling the dynamics and motions in action video; Deterministic models: dynamic time warping [25], maximum margin temporal warping [20], actom sequence model [6], graphs [3] and deep neural architectures [14, 17]; 8 / 57

32 Action Recognition Methods Handling Dynamics for Actions Focus on modeling the dynamics and motions in action video; Deterministic models: dynamic time warping [25], maximum margin temporal warping [20], actom sequence model [6], graphs [3] and deep neural architectures [14, 17]; Generative models: HMM [10], coupled HMM [2], CRF [22] and dynamic Bayes nets [24]. 8 / 57

33 Action Recognition Methods Kinect-Based Action Recognition and Detection Besides the RGB information, the depth information of action video is available from depth camera; 9 / 57

34 Action Recognition Methods Kinect-Based Action Recognition and Detection Besides the RGB information, the depth information of action video is available from depth camera; Specific feature descriptors and dynamic model for depth input are designed to alleviate the challenges of action recognition; 9 / 57

35 Action Recognition Methods Kinect-Based Action Recognition and Detection Besides the RGB information, the depth information of action video is available from depth camera; Specific feature descriptors and dynamic model for depth input are designed to alleviate the challenges of action recognition; Since depth information is available, it significantly reduces the difficulties in the estimation of human motion. The articulated human motion can be captured from the depth video. 9 / 57

36 Small Size Datasets The KTH Dataset [13] 10 / 57

37 Small Size Datasets The KTH Dataset [13] 6 actions (walking, jogging, running, boxing, hand waving and hand clapping) 10 / 57

38 Small Size Datasets The KTH Dataset [13] 6 actions (walking, jogging, running, boxing, hand waving and hand clapping) The Weizmann Dataset [1] 10 / 57

39 Small Size Datasets The KTH Dataset [13] 6 actions (walking, jogging, running, boxing, hand waving and hand clapping) The Weizmann Dataset [1] 10 actions (walk, run, jump, gallop sideways, bend, one-hand wave, two-hands wave, jump in place, jumping jack and skip) 10 / 57

40 Small Size Datasets The KTH Dataset [13] 6 actions (walking, jogging, running, boxing, hand waving and hand clapping) The Weizmann Dataset [1] 10 actions (walk, run, jump, gallop sideways, bend, one-hand wave, two-hands wave, jump in place, jumping jack and skip) The UCF Sports Action Dataset 10 / 57

41 Small Size Datasets The KTH Dataset [13] 6 actions (walking, jogging, running, boxing, hand waving and hand clapping) The Weizmann Dataset [1] 10 actions (walk, run, jump, gallop sideways, bend, one-hand wave, two-hands wave, jump in place, jumping jack and skip) The UCF Sports Action Dataset 9 actions (diving, golf swinging, kicking, weightlifting, horseback riding, running, skating, swinging a baseball bat and walking) 10 / 57

42 Large Size Datasets The IXMAS Dataset [23] 11 / 57

43 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) 11 / 57

44 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) Hollywood Human Action Dataset [11] 11 / 57

45 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) Hollywood Human Action Dataset [11] 12 actions (answer phone, get out of car, handshake, hug, kiss, sit down, sit up, stand up, drive car, eat, fight and run) 11 / 57

46 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) Hollywood Human Action Dataset [11] 12 actions (answer phone, get out of car, handshake, hug, kiss, sit down, sit up, stand up, drive car, eat, fight and run) The UCF50 Dataset [12]: 50 different actions/activities 11 / 57

47 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) Hollywood Human Action Dataset [11] 12 actions (answer phone, get out of car, handshake, hug, kiss, sit down, sit up, stand up, drive car, eat, fight and run) The UCF50 Dataset [12]: 50 different actions/activities The HMDB51 Dataset [8]: 51 different actions/activities 11 / 57

48 Large Size Datasets The IXMAS Dataset [23] 14 actions (check watch, cross arms, scratch head, sit down, get up, turn around, walk, wave, punch, kick, point, pick up, throw over head and throw from bottom up) Hollywood Human Action Dataset [11] 12 actions (answer phone, get out of car, handshake, hug, kiss, sit down, sit up, stand up, drive car, eat, fight and run) The UCF50 Dataset [12]: 50 different actions/activities The HMDB51 Dataset [8]: 51 different actions/activities The UCF101 Dataset [15]: 101 different actions/activities 11 / 57

49 Data Samples (a) KTH Dataset (b) Hollywood Dataset 12 / 57

50 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 13 / 57

51 Outline Introduction Modeling Space-Time Appearance Space-Time Interest Points (STIP) Recognizing Human Actions: A Local SVM Approach Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 14 / 57

52 Spatio-Temporal Interest Points 1 Motivated by the idea of Harris and Forstner spatial interest point operators, extended into the spatio-temporal domain; 1 I. Laptev. On space-time interest points. International Journal of Computer Vision, 64(2-3): , / 57

53 Spatio-Temporal Interest Points 1 Motivated by the idea of Harris and Forstner spatial interest point operators, extended into the spatio-temporal domain; Aim to find the good spatio-temporal positions in a sequence for feature extraction; 1 I. Laptev. On space-time interest points. International Journal of Computer Vision, 64(2-3): , / 57

54 Spatio-Temporal Interest Points 1 Motivated by the idea of Harris and Forstner spatial interest point operators, extended into the spatio-temporal domain; Aim to find the good spatio-temporal positions in a sequence for feature extraction; Distinct and stable descriptors are extracted from the obtained interest points; 1 I. Laptev. On space-time interest points. International Journal of Computer Vision, 64(2-3): , / 57

55 Spatio-Temporal Interest Points The points that have large variations along both the spatial and the temporal directions in local spatio temporal volumes. Figure : Detecting the strongest spatio-temporal interest points in a football sequence with a player heading the ball. 16 / 57

56 Spatio-Temporal Interest Point Detection In the spatial domain, we can model an image f sp by its linear scale-space representation L sp : L sp ( x, y; σl 2 ) = g sp ( x, y; σl 2 ) f sp (x, y) 17 / 57

57 Spatio-Temporal Interest Point Detection In the spatial domain, we can model an image f sp by its linear scale-space representation L sp : L sp ( x, y; σl 2 ) = g sp ( x, y; σl 2 ) f sp (x, y) Like the operation for image, we can model the sequence by a linear scale-space representation L: L ( ; σ 2 l, τ 2 l g ( x, y, t; σ 2 l, τ 2 l ) = g ( ; σ 2 l, τ 2 ) f ( ) l ) = exp( (x 2 + y 2 )/2σ 2 l t 2 /2τ 2 l ) (2π) 3 σ 4 l τ 2 l 17 / 57

58 Spatio-Temporal Interest Point Detection Construct a 3 3 spatio-temporal second-moment matrix: µ = g ( ; σi 2, τi 2 ) L 2 x L x L y L x L t L x L y L 2 y L y L t L x L t L y L t L 2 t 18 / 57

59 Spatio-Temporal Interest Point Detection Construct a 3 3 spatio-temporal second-moment matrix: µ = g ( ; σi 2, τi 2 ) L 2 x L x L y L x L t L x L y L 2 y L y L t L x L t L y L t L 2 t The first-order derivatives are defined as: (ξ = {x,y,t}) L ξ ( ; σ 2 l, τl 2 ) = ξ (g f ) 18 / 57

60 Spatio-Temporal Interest Point Detection Construct a 3 3 spatio-temporal second-moment matrix: µ = g ( ; σi 2, τi 2 ) L 2 x L x L y L x L t L x L y L 2 y L y L t L x L t L y L t L 2 t The first-order derivatives are defined as: (ξ = {x,y,t}) L ξ ( ; σ 2 l, τl 2 ) = ξ (g f ) Compute the three eigenvalues λ 1, λ 2 and λ 3 of µ, the Harris corner function is then defined as: H = det(µ) k trace 3 (µ) = λ 1 λ 2 λ 3 k(λ 1 + λ 2 + λ 3 ) 3 18 / 57

61 Spatio-Temporal Interest Point Detection Construct a 3 3 spatio-temporal second-moment matrix: µ = g ( ; σi 2, τi 2 ) L 2 x L x L y L x L t L x L y L 2 y L y L t L x L t L y L t L 2 t The first-order derivatives are defined as: (ξ = {x,y,t}) L ξ ( ; σ 2 l, τl 2 ) = ξ (g f ) Compute the three eigenvalues λ 1, λ 2 and λ 3 of µ, the Harris corner function is then defined as: H = det(µ) k trace 3 (µ) = λ 1 λ 2 λ 3 k(λ 1 + λ 2 + λ 3 ) 3 Detect the interest points by calculating the positive local maxima of H; 18 / 57

62 Space-Time Interest Points: Examples (a) Action : clapping hands (b) The detected interst points 19 / 57

63 Spatio-Temporal Scale Adaptation Let s recall the scale-space representation L ( ; σl 2, τ l 2 ), the two scale factors σl 2 and τl 2 influence the result a lot; 20 / 57

64 Spatio-Temporal Scale Adaptation Let s recall the scale-space representation L ( ; σl 2, τ l 2 ), the two scale factors σl 2 and τl 2 influence the result a lot; The larger the τl 2 is, the easier the space-time structures with long temporal extents are detected; 20 / 57

65 Spatio-Temporal Scale Adaptation Let s recall the scale-space representation L ( ; σl 2, τ l 2 ), the two scale factors σl 2 and τl 2 influence the result a lot; The larger the τl 2 is, the easier the space-time structures with long temporal extents are detected; The larger the σl 2 is, the easier the space-time structures with long spatial extents are detected; 20 / 57

66 Spatio-Temporal Scale Adaptation By finding the extrema of 2 norml over both spatial and temporal scales, we can automatically determine the scale. 21 / 57

67 Result Figure : STIP detection for a zoom-in sequence of a walking person. 22 / 57

68 Result Figure : (top): Correct matches in sequences with leg actions; (bottom): Correct matches in sequences with arm actions; 23 / 57

69 Outline Introduction Modeling Space-Time Appearance Space-Time Interest Points (STIP) Recognizing Human Actions: A Local SVM Approach Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 24 / 57

70 Recognition based on Local Space-time Features Figure : Local space-time features detected for a walking pattern 2 2 C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: a local svm approach. In Pattern Recognition, ICPR Proceedings of the 17th International Conference on, volume 3, pages IEEE, / 57

71 Representation of Features Spatial-temporal jets (4th order) are computed at each feature center: j = (L x, L y, L t, L xx,, L tttt ) σ 2 = σ 2 i,τ 2 = τ 2 i L x m y n t k = σm+n τ k ( x m y n t k g) f 26 / 57

72 Representation of Features Spatial-temporal jets (4th order) are computed at each feature center: j = (L x, L y, L t, L xx,, L tttt ) σ 2 = σ 2 i,τ 2 = τ 2 i L x m y n t k = σm+n τ k ( x m y n t k g) f Using k-means clustering over j, a vocabulary consisting of words h i is created from the jet descriptors; 26 / 57

73 Representation of Features Spatial-temporal jets (4th order) are computed at each feature center: j = (L x, L y, L t, L xx,, L tttt ) σ 2 = σ 2 i,τ 2 = τ 2 i L x m y n t k = σm+n τ k ( x m y n t k g) f Using k-means clustering over j, a vocabulary consisting of words h i is created from the jet descriptors; Finally, a given video is represented by a histogram of counts of occurrences of features corresponding to h i in that video: H = (h 1,..., h n ) 26 / 57

74 Recognition by Support Vector Machines For action recognition, the obtained local space-time features are used for SVM classification 27 / 57

75 Recognition by Support Vector Machines For action recognition, the obtained local space-time features are used for SVM classification Given a set of training data from different action classes {(H i, y i )} n i=1, a SVM classifier for each action class is learned: ( n ) f (H) = sgn α i y i H i + b i=1 27 / 57

76 Recognition by Support Vector Machines For action recognition, the obtained local space-time features are used for SVM classification Given a set of training data from different action classes {(H i, y i )} n i=1, a SVM classifier for each action class is learned: ( n ) f (H) = sgn α i y i H i + b i=1 Easy to extend to a kernelized version; 27 / 57

77 Results Figure : Results of action recognition for different methods and scenarios on KTH dataset. 28 / 57

78 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 29 / 57

79 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Coupled Hidden Markov Models for Complex Actions Action Recognition with Depth Cameras Action Detection 30 / 57

80 Coupled Hidden Markov Models 3 Hidden Markov model (HMM) is preferable for modeling and classifying dynamic behaviors. 3 M. Brand, N. Oliver, and A. Pentland. Coupled hidden markov models for complex action recognition. In Computer vision and pattern recognition, proceedings., 1997 ieee computer society conference on, pages IEEE, / 57

81 Coupled Hidden Markov Models 3 Hidden Markov model (HMM) is preferable for modeling and classifying dynamic behaviors. But HMM is not suitable for multiple interacting processes (have structure in both time and space). 3 M. Brand, N. Oliver, and A. Pentland. Coupled hidden markov models for complex action recognition. In Computer vision and pattern recognition, proceedings., 1997 ieee computer society conference on, pages IEEE, / 57

82 Coupled Hidden Markov Models 3 Hidden Markov model (HMM) is preferable for modeling and classifying dynamic behaviors. But HMM is not suitable for multiple interacting processes (have structure in both time and space). Coupled hidden Markov models can model multiple interacting processes without running afoul of the Markov condition. 3 M. Brand, N. Oliver, and A. Pentland. Coupled hidden markov models for complex action recognition. In Computer vision and pattern recognition, proceedings., 1997 ieee computer society conference on, pages IEEE, / 57

83 Limitations of HMM HMM is suitable for implicitly handling time-varying signals, which satisfy the Markov properties. 32 / 57

84 Limitations of HMM HMM is suitable for implicitly handling time-varying signals, which satisfy the Markov properties. But HMMs are not appropriate to model systems that have compositional state, e.g., multiple interacting processes that have structure in both time and space. 32 / 57

85 Limitations of HMM HMM is suitable for implicitly handling time-varying signals, which satisfy the Markov properties. But HMMs are not appropriate to model systems that have compositional state, e.g., multiple interacting processes that have structure in both time and space. Think about how to model A gave B the C? 32 / 57

86 Coupling and Factoring HMMs In order to handle multiple interacting processes (to couple HMMs), we need to obtain a joint HMM C from two coupling HMMs A and B; 33 / 57

87 Coupling and Factoring HMMs In order to handle multiple interacting processes (to couple HMMs), we need to obtain a joint HMM C from two coupling HMMs A and B; Given the states a i and b j and transition parameters P ai a j and P bk b l, the joint states is c ij = {a i, b j }, the transition is: P cik c jl ) = Ψ (P ai a j, P bk b l, P ai b l, P bk a j 33 / 57

88 Coupling and Factoring HMMs In order to handle multiple interacting processes (to couple HMMs), we need to obtain a joint HMM C from two coupling HMMs A and B; Given the states a i and b j and transition parameters P ai a j and P bk b l, the joint states is c ij = {a i, b j }, the transition is: P cik c jl ) = Ψ (P ai a j, P bk b l, P ai b l, P bk a j P ai b l and P bk a j are the coupling parameters; 33 / 57

89 Coupling and Factoring HMMs We can also project the joint HMM back into its components: P ai a j P ai b l l k j k P cik c jl P cik c jl 34 / 57

90 Coupling and Factoring HMMs We can also project the joint HMM back into its components: P ai a j l k P cik c jl P ai b l j k P cik c jl So a joint HMM can be trained via standard HMM methods; 34 / 57

91 One Example Figure : A CHMM is used to model to represent the action performed by two hands. 35 / 57

92 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 36 / 57

93 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Mining Actionlet Ensemble Action Detection 37 / 57

94 Depth Map and 3D Joints The figure shows some samples of the depth maps; 38 / 57

95 Depth Map and 3D Joints The figure shows some samples of the depth maps; the 3D joint position can be estimated or annotated from the depth maps; 38 / 57

96 Mining Actionlet Ensemble for Action Recognition 4 An actionlet ensemble model is learnt from depth maps to represent each action; 4 J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for action recognition with depth cameras. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages IEEE, / 57

97 Mining Actionlet Ensemble for Action Recognition 4 An actionlet ensemble model is learnt from depth maps to represent each action; Fourier Temporal Pyramid is proposed to represent the temporal dynamics; 4 J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for action recognition with depth cameras. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages IEEE, / 57

98 Mining Discriminative Actionlets An actionlet is defined as a subset of joints S: P S (y (j) = c x (j) ) = Π i S P i (y (j) = c x (j) ) 40 / 57

99 Mining Discriminative Actionlets An actionlet is defined as a subset of joints S: P S (y (j) = c x (j) ) = Π i S P i (y (j) = c x (j) ) For each joint, how discriminative it is can be defined as large confidence Conf S and small ambiguity Amb S ; Conf S = max j X c log P S (y (j) = c x (j) ) Amb S = j / X c log P S (y (j) = c x (j) ) 40 / 57

100 Mining Discriminative Actionlets An actionlet is defined as a subset of joints S: P S (y (j) = c x (j) ) = Π i S P i (y (j) = c x (j) ) For each joint, how discriminative it is can be defined as large confidence Conf S and small ambiguity Amb S ; Conf S = max j X c log P S (y (j) = c x (j) ) Amb S = j / X c log P S (y (j) = c x (j) ) Observation: adding a new joint into one actionlet will always reduce the confidence. 40 / 57

101 Mining Discriminative Actionlets An actionlet is defined as a subset of joints S: P S (y (j) = c x (j) ) = Π i S P i (y (j) = c x (j) ) For each joint, how discriminative it is can be defined as large confidence Conf S and small ambiguity Amb S ; Conf S = max j X c log P S (y (j) = c x (j) ) Amb S = j / X c log P S (y (j) = c x (j) ) Observation: adding a new joint into one actionlet will always reduce the confidence. The mining strategy is to keep the actionlet P c that Amb S T amb and Conf S T conf 40 / 57

102 Learning Actionlet Ensemble A multiple kernel learning approach is employed to learn an actionlet ensemble structure that combines these discriminative actionlets; 41 / 57

103 Learning Actionlet Ensemble A multiple kernel learning approach is employed to learn an actionlet ensemble structure that combines these discriminative actionlets; For each actionlet S k, an SVM model on it is defined as a linear output function: f k (x, y) = w k, Φ k (x, y) + b k 41 / 57

104 Learning Actionlet Ensemble A multiple kernel learning approach is employed to learn an actionlet ensemble structure that combines these discriminative actionlets; For each actionlet S k, an SVM model on it is defined as a linear output function: f k (x, y) = w k, Φ k (x, y) + b k A final output function is a convex combination of p kernels, each kernel corresponds to an actionlet: p f final (x, y) = [β k w k, Φ k (x, y) + b k ] k=1 41 / 57

105 Learning Actionlet Ensemble A multiple kernel learning approach is employed to learn an actionlet ensemble structure that combines these discriminative actionlets; For each actionlet S k, an SVM model on it is defined as a linear output function: f k (x, y) = w k, Φ k (x, y) + b k A final output function is a convex combination of p kernels, each kernel corresponds to an actionlet: p f final (x, y) = [β k w k, Φ k (x, y) + b k ] k=1 The resulting optimization problem can be solved by iteratively optimizing between β and w, b: 41 / 57

106 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection 42 / 57

107 Outline Introduction Modeling Space-Time Appearance Modeling Action Dynamics Action Recognition with Depth Cameras Action Detection Discriminative Subvolume Search for Efficient Action Detection 43 / 57

108 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns 44 / 57

109 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns They can be characterized by collections of spatio-temporal invariant features 44 / 57

110 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns They can be characterized by collections of spatio-temporal invariant features Detection of actions is to find the re-occurrences (e.g. through pattern matching) of such spatio-temporal patterns; 44 / 57

111 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns They can be characterized by collections of spatio-temporal invariant features Detection of actions is to find the re-occurrences (e.g. through pattern matching) of such spatio-temporal patterns; Two critical issues: 44 / 57

112 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns They can be characterized by collections of spatio-temporal invariant features Detection of actions is to find the re-occurrences (e.g. through pattern matching) of such spatio-temporal patterns; Two critical issues: searching for action in the video space is much more time consuming; 44 / 57

113 Difficulties in Action Detection Actions can be treated as spatio-temporal patterns They can be characterized by collections of spatio-temporal invariant features Detection of actions is to find the re-occurrences (e.g. through pattern matching) of such spatio-temporal patterns; Two critical issues: searching for action in the video space is much more time consuming; human actions involve tremendous intra-pattern variations. 44 / 57

114 Discriminative Subvolume Search for Efficient Action Detection 5 A discriminative pattern matching called naive Bayes based mutual information maximization (NBMIM) for multi-class action categorization; 5 J. Yuan, Z. Liu, and Y. Wu. Discriminative subvolume search for efficient action detection. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, / 57

115 Discriminative Subvolume Search for Efficient Action Detection 5 A discriminative pattern matching called naive Bayes based mutual information maximization (NBMIM) for multi-class action categorization; A novel search algorithm is proposed to locate the optimal subvolume in the 3D video space for efficient action detection; 5 J. Yuan, Z. Liu, and Y. Wu. Discriminative subvolume search for efficient action detection. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, / 57

116 The Proposed Idea Action detection is formulated as searching for a subvolume in video that has the maximum mutual information toward the action class; 46 / 57

117 The Proposed Idea Action detection is formulated as searching for a subvolume in video that has the maximum mutual information toward the action class; In the figure, each circle represents a spatio-temporal feature point which contributes a vote based on its own mutual information; 46 / 57

118 The Proposed Idea Action detection is formulated as searching for a subvolume in video that has the maximum mutual information toward the action class; In the figure, each circle represents a spatio-temporal feature point which contributes a vote based on its own mutual information; This is a new formulation for action detection! 46 / 57

119 Naive-Bayes Based Mutual Information Maximization (NBMIM) We represent an action by a collection of spatio-temporal interest points Q = {d i } (STIPs), then appearance feature HOG and motion feature HOF are extracted from each STIP; 47 / 57

120 Naive-Bayes Based Mutual Information Maximization (NBMIM) We represent an action by a collection of spatio-temporal interest points Q = {d i } (STIPs), then appearance feature HOG and motion feature HOF are extracted from each STIP; Evaluate the mutual information between a video clip Q and a specific class c: 47 / 57

121 Naive-Bayes Based Mutual Information Maximization (NBMIM) To evaluate the contribution s c (d q ) of each d q Q: 48 / 57

122 Naive-Bayes Based Mutual Information Maximization (NBMIM) To evaluate the contribution s c (d q ) of each d q Q: So the likelihood ratio test P(dq C c) P(d q C=c) of d q ; determines the property 48 / 57

123 Naive-Bayes Based Mutual Information Maximization (NBMIM) To evaluate the contribution s c (d q ) of each d q Q: So the likelihood ratio test P(dq C c) P(d q C=c) of d q ; For the C-class action categorization, we built C oneagainst-all classifiers as: determines the property 48 / 57

124 Action Detection in Video Based on the NBMIM criterion, given a video sequence V, the goal is to find a spatial-temporal subvolume (3D subvolume) V V : 49 / 57

125 Action Detection in Video Based on the NBMIM criterion, given a video sequence V, the goal is to find a spatial-temporal subvolume (3D subvolume) V V : Suppose the target video V is of size m n t, the total number of the 3D subvolumes is in the order of O(n 2 m 2 t 2 ), an exhaustive search is impossible! 49 / 57

126 Action Detection in Video Based on the NBMIM criterion, given a video sequence V, the goal is to find a spatial-temporal subvolume (3D subvolume) V V : Suppose the target video V is of size m n t, the total number of the 3D subvolumes is in the order of O(n 2 m 2 t 2 ), an exhaustive search is impossible! A new efficient search strategy is needed! 49 / 57

127 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: 50 / 57

128 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: An optimal subvolume V has 6 parameters, including top, bottom, left, right positions, start and end time; 50 / 57

129 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: An optimal subvolume V has 6 parameters, including top, bottom, left, right positions, start and end time; The complexity of the branch-and-bound grows exponentially in the number of dimensions; 50 / 57

130 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: An optimal subvolume V has 6 parameters, including top, bottom, left, right positions, start and end time; The complexity of the branch-and-bound grows exponentially in the number of dimensions; Our new efficient search: 50 / 57

131 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: An optimal subvolume V has 6 parameters, including top, bottom, left, right positions, start and end time; The complexity of the branch-and-bound grows exponentially in the number of dimensions; Our new efficient search: Instead of directly applying branch-and-bound in the 6D parameter space, our new method decomposes it into two subspaces: 4D spatial space and 2D temporal space; 50 / 57

132 Efficient Search for the Optimal 3D Subvolume Naive 3D branch-and-bound: An optimal subvolume V has 6 parameters, including top, bottom, left, right positions, start and end time; The complexity of the branch-and-bound grows exponentially in the number of dimensions; Our new efficient search: Instead of directly applying branch-and-bound in the 6D parameter space, our new method decomposes it into two subspaces: 4D spatial space and 2D temporal space; An optimal 3D volume V is determined by a spatial window W and a temporal segment T which has the maximum detection score: [W, T ] = arg max s(d) W,T d WxT 50 / 57

133 Efficient Search for the Optimal 3D Subvolume We take different search strategies in the two subspaces W and T and search alternately between W and T : 51 / 57

134 Efficient Search for the Optimal 3D Subvolume We take different search strategies in the two subspaces W and T and search alternately between W and T : Once W is determined, we search for the optimal temporal segment T. This is a 1D max subvector problem; 51 / 57

135 Efficient Search for the Optimal 3D Subvolume We take different search strategies in the two subspaces W and T and search alternately between W and T : Once W is determined, we search for the optimal temporal segment T. This is a 1D max subvector problem; To search the spatial parameter space W, we employ a branch-and-bound strategy with a tighter upper bound (Proved in the paper); 51 / 57

136 Experiment Result 52 / 57

137 References I M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions as space-time shapes. In Tenth IEEE International Conference on Computer Vision (ICCV 05) Volume 1, volume 2, pages IEEE, M. Brand, N. Oliver, and A. Pentland. Coupled hidden markov models for complex action recognition. In Computer vision and pattern recognition, proceedings., 1997 ieee computer society conference on, pages IEEE, W. Brendel and S. Todorovic. Learning spatiotemporal graphs of human activities. In 2011 International Conference on Computer Vision, pages IEEE, R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal. Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, G. Chéron, I. Laptev, and C. Schmid. P-cnn: Pose-based cnn features for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages , / 57

138 References II A. Gaidon, Z. Harchaoui, and C. Schmid. Actom sequence models for efficient action detection. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages IEEE, A. Klaser, M. Marsza lek, and C. Schmid. A spatio-temporal descriptor based on 3d-gradients. In BMVC th British Machine Vision Conference, pages British Machine Vision Association, H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: a large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages IEEE, I. Laptev. On space-time interest points. International Journal of Computer Vision, 64(2-3): , K. Li, J. Hu, and Y. Fu. Modeling complex temporal composition of actionlets for activity prediction. In European Conference on Computer Vision, pages Springer, / 57

139 References III M. Marszalek, I. Laptev, and C. Schmid. Actions in context. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, K. K. Reddy and M. Shah. Recognizing 50 human action categories of web videos. Machine Vision and Applications, 24(5): , C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: a local svm approach. In Pattern Recognition, ICPR Proceedings of the 17th International Conference on, volume 3, pages IEEE, K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pages , K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arxiv preprint arxiv: , N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using lstms. CoRR, abs/ , 2, / 57

140 References IV V. Veeriah, N. Zhuang, and G.-J. Qi. Differential recurrent neural networks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages , H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Action recognition by dense trajectories. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages IEEE, J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for action recognition with depth cameras. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages IEEE, J. Wang and Y. Wu. Learning maximum margin temporal warping for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages , L. Wang, Y. Qiao, and X. Tang. Action recognition with trajectory-pooled deep-convolutional descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , / 57

141 References V Y. Wang and G. Mori. Hidden part models for human action recognition: Probabilistic versus max margin. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7): , D. Weinland, E. Boyer, and R. Ronfard. Action recognition from arbitrary views using 3d exemplars. In 2007 IEEE 11th International Conference on Computer Vision, pages 1 7. IEEE, T. Xiang and S. Gong. Beyond tracking: Modelling activity and understanding behaviour. International Journal of Computer Vision, 67(1):21 51, B. Yao and S.-C. Zhu. Learning deformable action templates from cluttered videos. In 2009 IEEE 12th International Conference on Computer Vision, pages IEEE, J. Yuan, Z. Liu, and Y. Wu. Discriminative subvolume search for efficient action detection. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, / 57