arxiv: v1 [cs.cv] 9 Apr 2018
|
|
- Jodie Williamson
- 5 years ago
- Views:
Transcription
1 Fine-grained Activity Recognition in Baseball Videos AJ Piergiovanni and Michael S. Ryoo Department of Computer Science, Indiana University, Bloomington, IN arxiv: v1 [cs.cv] 9 Apr 2018 Abstract In this paper, we introduce a challenging new dataset, MLB-YouTube, designed for fine-grained activity detection. The dataset contains two settings: segmented video classification as well as activity detection in continuous videos. We experimentally compare various recognition approaches capturing temporal structure in activity videos, by classifying segmented videos and extending those approaches to continuous videos. We also compare models on the extremely difficult task of predicting pitch speed and pitch type from broadcast baseball videos. We find that learning temporal structure is valuable for fine-grained activity recognition. 1. Introduction Activity recognition is an important problem in computer vision with many applications within sports. Every major professional sporting event is recorded for entertainment purposes, but is also used for analysis by coaches, scouts, and media analysts. Many game statistics are currently manually tracked, but could be replaced by computer vision systems. Recently, the MLB has used the PITCHf/x and Statcast systems that are able to automatically capture pitch speed and motion. These systems use multiple highspeed cameras and radar to capture detailed measurements for every player on the field. However, much of this data is not publicly available. In this paper, we introduce a new dataset, MLB- YouTube, which contains densely annotated frames with activities from broadcast baseball videos. Unlike many existing activity recognition or detection datasets, ours focuses on fine-grained activity recognition. As shown in Fig. 1, the scene structure is very similar between activities, often the only difference is the motion of a single person. Additionally, we only have a single camera viewpoint to determine the activity. We experimentally compare various approaches for temporal feature pooling for both segmented video classification as well as activity detection in continuous videos. (a) (b) (c) Figure 1: Examples of (a) No swing, (b) Swing and (c) Bunting. This task is quite challenging as the difference between these activities is very small. 2. Related Works Activity recognition has been a popular research topic in computer vision [1, 10, 20, 25, 16]. Hand-crafted features, such as dense trajectories [25] gave promising results on many datasets. More recent works have focused on learning CNNs for activity recognition [3, 22]. Two-stream CNNs take spatial RGB frames and optical flow frames as input [20, 7]. 3D XYT convoltuional models have been trained to learn spatio-temporal features [22, 3, 23, 8]. To train these CNN models, large scale datasets such as Kinetics [11], THUMOS [9], and ActivityNet [6] have been created. Many works have explored temporal feature aggregation for activity recognition. Ng et al. [13] compared various pooling methods and found that LSTMs and max-pooling the entire video performed best. Ryoo et al. [17] found that pooling intervals of different locations/lengths was beneficial to activity recognition. Piergiovanni et al. [14] found that learning important sub-event intervals and using those for classification improved performance. Recently, segment-based 3D CNNs have been used to capture spatio-temporal information simultaneously for activity detection [26, 19, 18]. These approaches all rely on the 3D CNN to capture temporal dynamics, which usually only contain 16 frames. Some works have studied longerterm temporal structures [3, 10, 13, 24], but it was generally done with a temporal pooling of local representations or (spatio-)temporal convolutions with larger fixed intervals. Recurrent neural networks (RNNs) also have been used to model activity transitions between frames [27, 28, 5].
2 (a) (b) (c) (d) Figure 2: Examples of some of the activities in the MLB-YouTube Dataset. The activities are (a) Hit, (b) Bunt, (c) Hit by pitch, and (d) No activity (hard negatives). The difference between the activities is quite small, making this a challenging task. 3. MLB-YouTube Dataset We created a large-scale dataset consisting of 20 baseball games from the 2017 MLB post-season available on YouTube with over 42 hours of video footage. Our dataset consists of two components: segmented videos for activity recognition and continuous videos for activity classification. Our dataset is quite challenging as it is created from TV broadcast baseball games where multiple different activities share the camera angle. Further, the motion/appearance difference between the various activities is quite small (e.g., the difference between swinging the bat and bunting is very small), as shown in Fig. 2. Many existing activity detection datasets, such as THUMOS [9] and ActivityNet [6], contain a large variety of activities that vary in setting, scale, and camera angle. This makes even a single frame from one activity (e.g., swimming) to be very different from that of another activity (e.g., basketball). On the other hand, a single frame from one of our baseball videos is often not enough to classify the activity. Fig. 3 shows the small difference between a ball and strike. To distinguish these activities requires detecting if the batter swings or not, or detecting the umpire s signal (Fig. 4) for a strike, or no signal for a ball. Further complicating this task is that the umpire can be occluded by the batter or catcher and each umpire has a unique way to signal a strike. Our segmented video dataset consists of 4,290 video clips. Each clip is annotated with the various baseball activities that occur, such as swing, hit, ball, strike, foul, etc. A video clip can contain multiple activities, so we treat this as a multi-label classification task. A full list of the activities and the number of examples of each is shown in Table 1. We additionally annotated each clip containing a pitch with the pitch type (e.g., fastball, curveball, slider, etc.) and the speed of the pitch. We also collected a set of 2,983 hard negative examples where no action occurs. These examples (a) (b) Figure 3: The difference between a (a) strike and (b) ball is very small. Figure 4: Detecting a strike often relies on the umpire s signal (or lack of signal) at the end of the pitch sequence. include views of the crowd, the field, or the players standing before or after a pitch occurred. Examples of the activities and hard negatives are shown in Fig. 2. Our continuous video dataset consists of 2,
3 Table 1: The activity classes in the segmented MLB- YouTube dataset and the number of instances of the activity. Activity # Examples No Activity 2983 Ball 1434 Strike 1799 Swing 2506 Hit 1391 Foul 718 In Play 679 Bunt 24 Hit by Pitch 14 minute long clips from the videos. Each video frame is annotated with the baseball activities that occur. Each continuous clip contains on average of 7.2 activities, resulting in a total of over 15,000 activity instances. Our dataset and models are avaiable at 4. Segmented Video Recognition Approach We explore various methods of temporal feature aggregation for segmented video activity recognition. With segmented videos, the classification task is much easier as every frame (in the video) corresponds to the activity. The model does not need to determine when an activity begins and ends. The base component of our approaches is based on a CNN providing a per-frame (or per-segment) representation. We obtain this from standard two-stream CNNs [20, 7] using a recent deep CNNs such as I3D [3] or InceptionV3 [21]. Given v, the T D features from a video, where T is the temporal length of the video and D is the dimensionality of the feature, the standard method for feature pooling is maxor mean-pooling over the temporal dimension followed by a fully-connected layer to classify the video clip [13], as shown in Fig. 5(a). However, this provides only one representation for the entire video, and loses valuable temporal information. One way to address this is to use a fixed temporal pyramid of various lengths, as shown in Fig 5(b). We divide the input video into intervals of various lengths (1/2, 1/4, and 1/8), and max-pool each interval. We concatenate these pooled features together, resulting in a K D representation (K is the number of intervals in the temporal pyramid), and use a fully-connected layer to classify the clip. We also try learning temporal convolution filters, which can learn to aggregate local temporal structure. The kernel size is L 1 and it is applied to each frame. This allows each timestep representation to contain information from nearby frames. We then apply max-pooling over the output of the temporal convolution and use a fully-connected layer to classify the clip, shown in Fig. 5(c). While temporal pyramid pooling allows some structure to be preserved, the intervals are predetermined and fixed. Previous works have found learning the sub-interval to pool was beneficial to activity recognition [14]. The learned intervals are controlled by 3 learned parameters, a center g, a width σ and a stride δ used to parameterize N Gaussians. Given T, the length of the video, we first compute the locations of the strided Gaussians as: g n = 0.5 T ( g n + 1) δ n = T N 1 δ n µ i n = g n + (i 0.5N + 0.5)δ n The filters are then created as: F m [i, t] = 1 exp( (t µi m) 2 ) Z m 2σ 2 m i {0, 1,..., N 1}, t {0, 1,..., T 1} where Z m is a normalization constant. We apply F to the T D video representation by matrix multiplication, resulting in a N D representation which is used as input to a fully connected layer for classification. This method is shown in Fig 5(d). Other works have used LSTMs [13, 4] to model temporal structure in videos. We also compare to a bi-directional LSTM with 512 hidden units where we use the last hidden state as input to a fully-connected layer for classification. We formulate our tasks as multi-label classification and train these models to minimize binary cross entropy: L(v) = z c log(p(c G(v)))+(1 z c ) log(1 p(c G(v))) c (3) Where G(v) is the function that pools the temporal information (i.e., max-pooling, LSTM, temporal convolution, etc.), and z c is the ground truth label for class c. 5. Activity Detection in Continuous Videos Activity detection in continuous videos is a more challenging problem. Here, our objective is to classify each frame with the occurring activities. Unlike segmented videos, there are multiple instances of activities occurring sequentially, often separated by frames with no activity. This requires the model to learn to detect the start and end of activities. As a baseline, we train a single fully-connected layer as a per-frame classifier. This method uses no temporal information not present in the features. We extend the approaches presented for segmented video classification to continuous videos by applying each approach in a temporal sliding window fashion. To do this, we first pick a fixed window duration (i.e., a temporal window of L features). We apply max-pooling to each window (as in Fig. 5(a)) and classify each pooled segment. (1) (2)
4 1xD 1xD KxD Lx1 NxD (a) (b) (c) (d) Figure 5: Illustration of the various feature aggregation methods. (a) Temporal max/mean-pooling, (b) Temporal Pyramid Pooling, (c) Temporal convolution followed by temporal max-pooling, (d) Sub-events. All models are followed by a fullyconnected layer for classification. We can similarly extend temporal pyramid pooling. Here, we split the window of length L into segments of length L/2, L/4, L/8, this results in 14 segments for each window. We apply max-pooling to each segment and concatenate the pooled features together. This gives a 14 D- dim representation for each window which is used as input to the classifier. For temporal convolutional models on continuous videos, we slightly alter the segmented video approach. Here, we learn a temporal convolutional kernel of length L and convolve it with the input video features. This operation takes input of size T D and produces output of size T D. We then apply a per-frame classifier on this representation. This allows the model to learn to aggregate local temporal information. To extend the sub-event model to continuous videos, we follow the approach above, but set T = L in Eq. 1. This results in filters of length L. Given v, the T D video representation, we convolve (instead of using matrix multiplication) the sub-event filters, F, with the input, resulting in a N D T -dim representation. We use this as input to a fully-connected layer to classify each frame. We train the model to minimize the per-frame binary classification: L(v) = t,c z t,c log(p(c H(v t )))+ (1 z t,c ) log(1 p(c H(v t ))) where v t is the per-frame or per-segment feature at time t, H(v t ) is the sliding window application of one of the feature pooling methods, and z t,c is the ground truth class at time t. A recent approach to learn super-events (i.e., global video context) was proposed and found to be effective for (4) activity detection in continuous videos [15]. The approach learns a set of temporal structure filters that are modeled as a set of N Cauchy distributions. Each distribution learns a center, x n and a width, γ n. Given T, the length of the video, the filters are constructed by: xˆ n = (T 1) (tanh (x n) + 1) 2 γˆ n = exp(1 2 tanh (γ n ) ) F [t, n] = 1 xn ) ( (t ˆ Z n πγˆ n ˆ γ n ) 2 (5) where Z n is a normalization constant, t {1, 2,..., T } and n {1, 2,..., N}. The filters are combined with learned per-class softattention weights, A and the super-event representation is computed as: S c = M A c,m m T F m [t] v t (6) where v is the T D video representation. These filters allow the model to learn intervals to focus on for useful temporal context. The super-event representation is concatenated to each timestep and used for classification. We also try concatenating the super- and sub-event representations to use for classification to create a three-level hierarchy of event representation. t
5 Table 2: Results on segmented videos performing binary pitch/non-pitch classification. Model RGB Flow Two-stream InceptionV InceptionV3 + sub-events I3D I3D + sub-events Experiments 6.1. Implementation Details As our base per-segment CNN, we use the I3D [3] network pretrained on the ImageNet and Kinetics [11] datasets. I3D obtained state-of-the-art results on segmented video tasks, and this allows us to obtain reliable per-segment feature representation. We also use two-stream version of InceptionV3 [21] pretrained on Imagenet and Kinetics as our base per-frame CNN, and compared them. We chose InceptionV3 as it is deeper than previous two-stream CNNs such as [20, 7]. We extracted frames from the videos at 25 fps, computed TVL1 [29] optical flow, clipped to [ 20, 20]. For InceptionV3, we computed features for every 3 frames (8 fps). For I3D, every frame was used as the input. I3D has a temporal stride of 8, resulting in 3 features per second (3 fps). We implemented the models in PyTorch. We trained our models using the Adam [12] optimizer with the learning rate set to We decayed the learning rate by a factor of 0.1 after every 10 training epochs. We trained our models for 50 epochs. Our source code, dataset and trained models are available at Segmented Video Activity Recognition We first performed the binary pitch/non-pitch classification of each video segment. This task is relatively easy, as the difference between pitch frames and non-pitch frames are quite different. The results, shown in Table 2, do not show much difference between the various features or models Multi-label Classification We evaluate and compare the various approaches of temporal feature aggregation by computing mean average precision (map) for each video clip, which is a standard evaluation metric for multi-label classification tasks. Table 4 compares the performance of the various temporal feature pooling methods. We find that all approaches outperform mean/max-pooling, confirming that maintaining some temporal structure is important for activity recognition. We find that fixed temporal pyramid pooling and LSTMs give some improvement. Temporal convolution provides a larger increase in performance, however it requires significantly Table 3: Additional number of parameters for models when added to base (e.g., I3D or Inception V3). Model # Parameters Max/Mean Pooling 16K Pyramid Pooling 115K LSTM 10.5M Temporal Conv 31.5M Sub-events 36K Table 4: map results on segmented videos performing multi-label classification. We find that learning subintervals to pool is important for activity recognition. Method RGB Flow Two-stream Random InceptionV3 + mean-pool InceptionV3 + max-pool InceptionV3 + pyramid InceptionV3 + LSTM InceptionV3 + temporal conv InceptionV3 + sub-events I3D + mean-pool I3D + max-pool I3D + pyramid I3D + LSTM I3D + temporal conv I3D + sub-events more parameters (see Table 3). Learning sub-events of [14] we found to give the best performance on this task. While LSTMs and temporal have been previously used for this task, they require greater number of parameters and perform worse, likely due to overfitting. Additionally, LSTMs require the video features to be processes sequentially as each timestep requires the output from the previous timestep, while the other approaches can be completely parallelized. In Table 5, we compare the average precision for each activity class. Learning temporal structure is especially helpful for frame-based features (e.g., InceptionV3) whose features capture minimal temporal information when compared to segment-based features (e.g., I3D) which capture some temporal information. Additionally, we find that subevent learning helps especially in the case of strikes, hits, foul balls, and hit by pitch, as those all have changes in video features after the event. For example, after the ball is hit, the camera will often follow the ball s trajectory, while being hit by a pitch the camera will follow the player walking to first base, as shown in Fig. 6 and Fig Pitch Speed Regression Pitch speed regression from video frames is a challenging task because it requires the network to learn localize the
6 Figure 6: Illustration of the learned sub-events for the hit by pitch activity. It localizes the start of the pitch as well as the batter walking towards first base. Figure 7: Illustration of the learned sub-event for the hit activity. It captures the start of the pitch as well as the camera change to watch the runner and ball in the field. start of a pitch and the end of the pitch, then compute the speed from a weak signal (i.e., only pitch speed). The baseball is often small and occluded by the pitcher. Professional baseball pitchers can throw the ball in excess of 100mph and the pitch only travels 60.5 ft. Thus the ball is only traveling for roughly 0.5 seconds. Using our initial frame rates of 8fps and 3fps, there was only 1-2 features of the pitch in the air, which we found was not enough to determine pitch speed. The YouTube videos contain 60fps, so we recomputed optical flow and extract RGB frames at 60fps. We use a fully-connected layer with one output to predict the pitch speed and minimize the L 1 loss between the ground truth speed and predicted speed. Using features extracted at 60fps, we were able to determine pitch speed, with 3.6mph average error. Table 6 compares various models. Fig. 8 shows the sub-event learned for various speeds Pitch Type Classification We experiment to see if it is possible to predict the pitch type from video. This is an extremely challenging problem because it is adversarial; pitchers practice to disguise their pitch from batters. Additionally, the difference between pitches can be as small as a difference in grip on the ball and which way it rotates with respect to the laces, which is rarely visible in broadcast baseball videos. In addition to the video features used in the previous experiments, we also extract pose using OpenPose [2]. Our features are heatmaps of joint and body part locations which we stack along the channel axis and use as input to an InceptionV3 CNN which we newly train on this task. We chose to try pose features as the body mechanics can vary between pitches as well (e.g., the stride length and arm angles can be different for fastballs and curveballs). Our dataset has 6 different pitches (fastball, sinker, curveball, changeup, slider, and knuckle-curve). We report our results in Table 7. We find that LSTMs actually perform worse than the baseline, likely due to overfitting the small differences between pitch types, while learning subevents helps. We observe that fastballs are the easiest to detect (68% accuracy) followed by sliders (45% accuracy), while sinkers are the hardest to classify (12%) Continuous Video Activity Detection We evaluate the extended models on continuous videos using per-frame mean average precision (map), and the results are shown in Table 8. This setting is more challenging that the segmented videos as the model must determine when the activity starts and ends and contains negative examples that are more ambiguous than the hard negatives in the segmented dataset (e.g., the model has to determine when the pitch event begins compared to just the pitcher standing on the mound). We find that all models improve over the baseline per-frame classification, confirming that temporal information is important for detection. We find that fixed temporal pyramid pooling outperforms max-pooling. The LSTM and temporal convolution seem to overfit, due to the larger number of parameters. We find that the convolutional form of sub-events to pool local temporal structure especially helps frame based features, not as much on segment features. Using the super-event approach [15], further improves performance. Combining the convolutional sub-event representation with the super-event
7 Table 5: Per-class average precision for segmented videos performing multi-label activity classification using two-stream features. We find that using sub-events to learn temporal intervals of interest is beneficial to activity recognition. Method Ball Strike Swing Hit Foul In Play Bunt Hit by Pitch Random InceptionV3 + max-pool InceptionV3 + sub-events I3D + max-pool I3D + sub-events Figure 8: Illustration of the learned sub-events used for the pitch speed regression task. Each sub-event localizes different start/end times for the pitch and this information is used to predict the pitch speed. Table 6: Results on segmented video pitch speed regression. We are reporting their root-mean-squared errors. Method I3D I3D + LSTM I3D + sub-events InceptionV3 InceptionV3 + LSTM InceptionV3 + sub-events Two-stream 4.3 mph 4.1 mph 3.9 mph 5.3 mph 4.5 mph 3.6 mph Table 7: Accuracy of our pitch type classification using I3D on video inputs and InceptionV3 trained using pose heatmaps as input. Method Accuracy Random 17.0% I3D 25.8% I3D + LSTM 18.5% I3D + sub-events 34.5% Pose 28.4% Pose + LSTM 27.6% Pose + sub-events 36.4% representation provides the best performance. 7. Conclusion We introduced a challenging new dataset, MLB- YouTube, for fine-grained activity recognition in videos. Table 8: Results on continuous videos performing multilabel activity classification (per-frame map). Method RGB Flow Two-stream Random I3D I3D + max-pooling I3D + pyramid I3D + LSTM I3D + temporal conv I3D + sub-events I3D + super-events I3D + sub+super-events InceptionV InceptionV3 + max-pooling InceptionV3 + pyramid InceptionV3 + LSTM InceptionV3 + temporal conv InceptionV3 + sub-events InceptionV3 + super-events InceptionV3 + sub+super-events We experimentally compare various recognition approaches with temporal feature pooling for both segmented and continuous videos. We find that learning sub-events to select the temporal regions-of-interest provides the best performance for segmented video classification. For detection in continuous videos, we find that learning convolutional sub-events combined with the super-event representation to form a three-level activity hierarchy provides the best performance.
8 References [1] J. K. Aggarwal and M. S. Ryoo. Human activity analysis: A review. ACM Computing Surveys, 43:16:1 16:43, April [2] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multiperson 2d pose estimation using part affinity fields. In CVPR, [3] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), , 3, 5 [4] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [5] V. Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem. Daps: Deep action proposals for action understanding. In Proceedings of European Conference on Computer Vision (ECCV), pages Springer, [6] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , , 2 [7] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , , 3, 5 [8] K. Hara, H. Kataoka, and Y. Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the ICCV Workshop on Action, Gesture, and Emotion Recognition, volume 2, page 4, [9] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with a large number of classes , 2 [10] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [11] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arxiv preprint arxiv: , , 5 [12] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arxiv preprint arxiv: , [13] J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages IEEE, , 3 [14] A. Piergiovanni, C. Fan, and M. S. Ryoo. Learning latent sub-events in activity videos using temporal attention filters. In Proceedings of the American Association for Artificial Intelligence (AAAI), , 3, 5 [15] A. Piergiovanni and M. S. Ryoo. Learning latent superevents to detect multiple activities in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), , 6 [16] M. S. Ryoo and L. Matthies. First-person activity recognition: What are they doing to me? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [17] M. S. Ryoo, B. Rothrock, and L. Matthies. Pooled motion features for first-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [18] Z. Shou, J. Chan, A. Zareian, K. Miyazawa, and S.-F. Chang. Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. arxiv preprint arxiv: , [19] Z. Shou, D. Wang, and S.-F. Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [20] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems (NIPS), pages , , 3, 5 [21] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , , 5 [22] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri. C3d: generic features for video analysis. CoRR, abs/ , 2(7):8, [23] D. Tran, J. Ray, Z. Shou, S.-F. Chang, and M. Paluri. Convnet architecture search for spatiotemporal feature learning. arxiv preprint arxiv: , [24] G. Varol, I. Laptev, and C. Schmid. Long-term Temporal Convolutions for Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, [25] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Action recognition by dense trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages IEEE, [26] H. Xu, A. Das, and K. Saenko. R-c3d: Region convolutional 3d network for temporal activity detection. arxiv preprint arxiv: , [27] S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei. Every moment counts: Dense detailed labeling of actions in complex videos. International Journal of Computer Vision (IJCV), pages 1 15, [28] S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei. Endto-end learning of action detection from frame glimpses in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages ,
9 [29] C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime tv-l 1 optical flow. In Joint Pattern Recognition Symposium, pages Springer,
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh National Institute of Advanced Industrial Science and Technology (AIST) Tsukuba,
More informationarxiv: v2 [cs.cv] 2 Apr 2018
Depth of 3D CNNs Depth of 2D CNNs Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? arxiv:1711.09577v2 [cs.cv] 2 Apr 2018 Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh National Institute
More informationLearning Latent Sub-events in Activity Videos Using Temporal Attention Filters
Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters AJ Piergiovanni, Chenyou Fan, and Michael S Ryoo School of Informatics and Computing, Indiana University, Bloomington, IN
More informationTemporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks Alberto Montes al.montes.gomez@gmail.com Santiago Pascual TALP Research Center santiago.pascual@tsc.upc.edu Amaia Salvador
More informationarxiv: v1 [cs.cv] 26 Jul 2018
A Better Baseline for AVA Rohit Girdhar João Carreira Carl Doersch Andrew Zisserman DeepMind Carnegie Mellon University University of Oxford arxiv:1807.10066v1 [cs.cv] 26 Jul 2018 Abstract We introduce
More information4D Effect Video Classification with Shot-aware Frame Selection and Deep Neural Networks
4D Effect Video Classification with Shot-aware Frame Selection and Deep Neural Networks Thomhert S. Siadari 1, Mikyong Han 2, and Hyunjin Yoon 1,2 Korea University of Science and Technology, South Korea
More informationarxiv: v2 [cs.cv] 6 May 2018
Appearance-and-Relation Networks for Video Classification Limin Wang 1,2 Wei Li 3 Wen Li 2 Luc Van Gool 2 1 State Key Laboratory for Novel Software Technology, Nanjing University, China 2 Computer Vision
More informationMiCT: Mixed 3D/2D Convolutional Tube for Human Action Recognition
MiCT: Mixed 3D/2D Convolutional Tube for Human Action Recognition Yizhou Zhou 1 Xiaoyan Sun 2 Zheng-Jun Zha 1 Wenjun Zeng 2 1 University of Science and Technology of China 2 Microsoft Research Asia zyz0205@mail.ustc.edu.cn,
More informationSpotlight: A Smart Video Highlight Generator Stanford University CS231N Final Project Report
Spotlight: A Smart Video Highlight Generator Stanford University CS231N Final Project Report Jun-Ting (Tim) Hsieh junting@stanford.edu Chengshu (Eric) Li chengshu@stanford.edu Kuo-Hao Zeng khzeng@cs.stanford.edu
More informationAggregating Frame-level Features for Large-Scale Video Classification
Aggregating Frame-level Features for Large-Scale Video Classification Shaoxiang Chen 1, Xi Wang 1, Yongyi Tang 2, Xinpeng Chen 3, Zuxuan Wu 1, Yu-Gang Jiang 1 1 Fudan University 2 Sun Yat-Sen University
More informationarxiv: v1 [cs.cv] 23 Jan 2018
Let s Dance: Learning From Online Dance Videos Daniel Castro Georgia Institute of Technology Steven Hickson Patsorn Sangkloy shickson@gatech.edu patsorn sangkloy@gatech.edu arxiv:1801.07388v1 [cs.cv] 23
More informationMultimodal Keyless Attention Fusion for Video Classification
The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) Multimodal Keyless Attention Fusion for Video Classification Xiang Long, 1 Chuang Gan, 1 Gerard de Melo, 2 Xiao Liu, 3 Yandong Li,
More informationarxiv: v1 [cs.cv] 26 Nov 2018
Evolving Space-Time Neural Architectures for Videos AJ Piergiovanni, Anelia Angelova, Alexander Toshev, Michael S. Ryoo Google Brain {ajpiergi,anelia,toshev,mryoo}@google.com arxiv:8.066v [cs.cv] 26 Nov
More informationarxiv: v1 [cs.cv] 14 Jul 2017
Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding Fu Li, Chuang Gan, Xiao Liu, Yunlong Bian, Xiang Long, Yandong Li, Zhichao Li, Jie Zhou, Shilei Wen Baidu IDL & Tsinghua University
More informationTemporal Difference Networks for Video Action Recognition
Temporal Difference Networks for Video Action Recognition Joe Yue-Hei Ng Larry S. Davis University of Maryland, College Park {yhng,lsd}@umiacs.umd.edu Abstract Deep convolutional neural networks have been
More informationarxiv: v3 [cs.cv] 12 Apr 2018
A Closer Look at Spatiotemporal Convolutions for Action Recognition Du Tran 1, Heng Wang 1, Lorenzo Torresani 1,2, Jamie Ray 1, Yann LeCun 1, Manohar Paluri 1 1 Facebook Research 2 Dartmouth College {trandu,hengwang,torresani,jamieray,yann,mano}@fb.com
More informationarxiv: v1 [cs.cv] 22 Nov 2017
D Nets: New Architecture and Transfer Learning for Video Classification Ali Diba,4,, Mohsen Fayyaz,, Vivek Sharma, Amir Hossein Karami 4, Mohammad Mahdi Arzani 4, Rahman Yousefzadeh 4, Luc Van Gool,4 ESAT-PSI,
More informationT-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition
The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition Kun Liu, 1 Wu Liu, 1 Chuang Gan, 2 Mingkui Tan, 3 Huadong
More informationGPU Accelerated Sequence Learning for Action Recognition. Yemin Shi
GPU Accelerated Sequence Learning for Action Recognition Yemin Shi shiyemin@pku.edu.cn 2018-03 1 Background Object Recognition (Image Classification) Action Recognition (Video Classification) Action Recognition
More informationEvaluation of Triple-Stream Convolutional Networks for Action Recognition
Evaluation of Triple-Stream Convolutional Networks for Action Recognition Dichao Liu, Yu Wang and Jien Kato Graduate School of Informatics Nagoya University Nagoya, Japan Email: {liu, ywang, jien} (at)
More informationQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset. By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018 Outline: Introduction Action classification architectures
More informationDeep Local Video Feature for Action Recognition
Deep Local Video Feature for Action Recognition Zhenzhong Lan 1 Yi Zhu 2 Alexander G. Hauptmann 1 Shawn Newsam 2 1 Carnegie Mellon University 2 University of California, Merced {lanzhzh,alex}@cs.cmu.edu
More informationarxiv: v3 [cs.cv] 2 Aug 2017
Action Detection ( 4.3) Tube Proposal Network ( 4.1) Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos Rui Hou, Chen Chen, Mubarak Shah Center for Research in Computer Vision (CRCV),
More informationA Unified Method for First and Third Person Action Recognition
A Unified Method for First and Third Person Action Recognition Ali Javidani Department of Computer Science and Engineering Shahid Beheshti University Tehran, Iran a.javidani@mail.sbu.ac.ir Ahmad Mahmoudi-Aznaveh
More informationVideo Imprint Segmentation for Temporal Action Detection in Untrimmed Videos
Video Imprint Segmentation for Temporal Action Detection in Untrimmed Videos Zhanning Gao 1,2, Le Wang 1, Qilin Zhang 3, Zhenxing Niu 2, Nanning Zheng 1, Gang Hua 4 1 Xi an Jiaotong University, 2 Alibaba
More informationDeep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon
Deep Learning For Video Classification Presented by Natalie Carlebach & Gil Sharon Overview Of Presentation Motivation Challenges of video classification Common datasets 4 different methods presented in
More informationMultilayer and Multimodal Fusion of Deep Neural Networks for Video Classification
Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification Xiaodong Yang, Pavlo Molchanov, Jan Kautz INTELLIGENT VIDEO ANALYTICS Surveillance event detection Human-computer interaction
More informationHierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network
Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network Feng Mao [0000 0001 6171 3168], Xiang Wu [0000 0003 2698 2156], Hui Xue, and Rong Zhang Alibaba Group, Hangzhou, China
More informationTowards Good Practices for Multi-modal Fusion in Large-scale Video Classification
Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification Jinlai Liu, Zehuan Yuan, and Changhu Wang Bytedance AI Lab {liujinlai.licio,yuanzehuan,wangchanghu}@bytedance.com Abstract.
More informationEasyChair Preprint. Real-Time Action Recognition based on Enhanced Motion Vector Temporal Segment Network
EasyChair Preprint 730 Real-Time Action Recognition based on Enhanced Motion Vector Temporal Segment Network Xue Bai, Enqing Chen and Haron Chweya Tinega EasyChair preprints are intended for rapid dissemination
More informationarxiv: v7 [cs.cv] 21 Apr 2018
End-to-end Video-level Representation Learning for Action Recognition Jiagang Zhu 1,2, Wei Zou 1, Zheng Zhu 1,2 1 Institute of Automation, Chinese Academy of Sciences (CASIA) 2 University of Chinese Academy
More informationActivity Recognition in Temporally Untrimmed Videos
Activity Recognition in Temporally Untrimmed Videos Bryan Anenberg Stanford University anenberg@stanford.edu Norman Yu Stanford University normanyu@stanford.edu Abstract We investigate strategies to apply
More informationA Torch Library for Action Recognition and Detection Using CNNs and LSTMs
A Torch Library for Action Recognition and Detection Using CNNs and LSTMs Gary Thung and Helen Jiang Stanford University {gthung, helennn}@stanford.edu Abstract It is very common now to see deep neural
More informationarxiv: v1 [cs.cv] 30 May 2017
Trampoline ing Trampoline ing Generic Tubelet Proposals for Action Localization arxiv:1705.10861v1 [cs.cv] 30 May 2017 Jiawei He Simon Fraser University jha203@sfu.ca Abstract We develop a novel framework
More informationDepth-Based Activity Recognition in ICUs Using Convolutional and Recurrent Neural Networks
Depth-Based Activity Recognition in ICUs Using Convolutional and Recurrent Neural Networks Rishab Mehra Gabriel M. Bianconi Serena Yeung Li Fei-Fei Department of Computer Science, Stanford University {rishab,bianconi,serena,feifeili}@cs.stanford.edu
More informationarxiv: v2 [cs.cv] 13 Jun 2017
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang arxiv:1703.01515v2
More informationarxiv: v1 [cs.cv] 29 Apr 2016
Improved Dense Trajectory with Cross Streams arxiv:1604.08826v1 [cs.cv] 29 Apr 2016 ABSTRACT Katsunori Ohnishi Graduate School of Information Science and Technology University of Tokyo ohnishi@mi.t.utokyo.ac.jp
More informationLarge-scale Video Classification with Convolutional Neural Networks
Large-scale Video Classification with Convolutional Neural Networks Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei Note: Slide content mostly from : Bay Area
More informationarxiv: v2 [cs.cv] 4 Aug 2017
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals Jiyang Gao 1 Zhenheng Yang 1 Chen Sun 2 Kan Chen 1 Ram Nevatia 1 1 University of Southern California 2 Google Research {jiyangga,
More informationCS231N Section. Video Understanding 6/1/2018
CS231N Section Video Understanding 6/1/2018 Outline Background / Motivation / History Video Datasets Models Pre-deep learning CNN + RNN 3D convolution Two-stream What we ve seen in class so far... Image
More informationYouTube-8M Video Classification
YouTube-8M Video Classification Alexandre Gauthier and Haiyu Lu Stanford University 450 Serra Mall Stanford, CA 94305 agau@stanford.edu hylu@stanford.edu Abstract Convolutional Neural Networks (CNNs) have
More informationStNet: Local and Global Spatial-Temporal Modeling for Action Recognition
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition Dongliang He 1 Zhichao Zhou 1 Chuang Gan 2 Fu Li 1 Xiao Liu 1 Yandong Li 3 Limin Wang 4 Shilei Wen 1 Department of Computer Vision
More informationTube Convolutional Neural Network (T-CNN) for Action Detection in Videos
Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos Rui Hou, Chen Chen, Mubarak Shah Center for Research in Computer Vision (CRCV), University of Central Florida (UCF) houray@gmail.com,
More informationarxiv: v1 [cs.cv] 15 Apr 2016
arxiv:0.09v [cs.cv] Apr 0 Long-term Temporal Convolutions for Action Recognition Gül Varol, Ivan Laptev Cordelia Schmid Inria Abstract. Typical human actions last several seconds and exhibit characteristic
More informationProceedings of the International MultiConference of Engineers and Computer Scientists 2018 Vol I IMECS 2018, March 14-16, 2018, Hong Kong
, March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong TABLE I CLASSIFICATION ACCURACY OF DIFFERENT PRE-TRAINED MODELS ON THE TEST DATA
More informationLong-term Temporal Convolutions for Action Recognition
1 Long-term Temporal Convolutions for Action Recognition Gül Varol, Ivan Laptev, and Cordelia Schmid, Fellow, IEEE arxiv:1604.04494v2 [cs.cv] 2 Jun 2017 Abstract Typical human actions last several seconds
More informationAttention-based Temporal Weighted Convolutional Neural Network for Action Recognition
Attention-based Temporal Weighted Convolutional Neural Network for Action Recognition Jinliang Zang 1, Le Wang 1, Ziyi Liu 1, Qilin Zhang 2, Zhenxing Niu 3, Gang Hua 4, and Nanning Zheng 1 1 Xi an Jiaotong
More informationA Proposal-Based Solution to Spatio-Temporal Action Detection in Untrimmed Videos
A Proposal-Based Solution to Spatio-Temporal Action Detection in Untrimmed Videos not posses these challenges. Hence, algorithms trained on these datasets have sub-optimal performance on untrimmed surveillance
More informationChannel Locality Block: A Variant of Squeeze-and-Excitation
Channel Locality Block: A Variant of Squeeze-and-Excitation 1 st Huayu Li Northern Arizona University Flagstaff, United State Northern Arizona University hl459@nau.edu arxiv:1901.01493v1 [cs.lg] 6 Jan
More informationarxiv: v2 [cs.cv] 26 Apr 2018
Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition arxiv:1804.07187v2 [cs.cv] 26 Apr 2018 Okan Köpüklü Neslihan Köse Gerhard Rigoll Institute for Human-Machine Communication Technical
More informationP-CNN: Pose-based CNN Features for Action Recognition. Iman Rezazadeh
P-CNN: Pose-based CNN Features for Action Recognition Iman Rezazadeh Introduction automatic understanding of dynamic scenes strong variations of people and scenes in motion and appearance Fine-grained
More informationCTAP: Complementary Temporal Action Proposal Generation
CTAP: Complementary Temporal Action Proposal Generation Jiyang Gao, Kan Chen, Ram Nevatia University of Southern California {jiyangga, kanchen, nevatia}@usc.edu Abstract. Temporal action proposal generation
More informationarxiv: v2 [cs.cv] 8 May 2018
J. GAO: REVISITING TEMPORAL MODELING FOR VIDEO-BASED PERSON REID 1 arxiv:1805.02104v2 [cs.cv] 8 May 2018 Revisiting Temporal Modeling for Video-based Person ReID Jiyang Gao jiyangga@usc.edu Ram Nevatia
More informationarxiv: v1 [cs.cv] 26 Apr 2018
Deep Keyframe Detection in Human Action Videos Xiang Yan1,2, Syed Zulqarnain Gilani2, Hanlin Qin1, Mingtao Feng3, Liang Zhang4, and Ajmal Mian2 arxiv:1804.10021v1 [cs.cv] 26 Apr 2018 1 School of Physics
More informationEigen-Evolution Dense Trajectory Descriptors
Eigen-Evolution Dense Trajectory Descriptors Yang Wang, Vinh Tran, and Minh Hoai Stony Brook University, Stony Brook, NY 11794-2424, USA {wang33, tquangvinh, minhhoai}@cs.stonybrook.edu Abstract Trajectory-pooled
More informationarxiv: v1 [cs.cv] 30 Dec 2018
Actor Conditioned Attention Maps for Video Action Detection arxiv:1812.11631v1 [cs.cv] 30 Dec 2018 Oytun Ulutan University of California, Santa Barbara Mudhakar Srivatsa IBM T. J. Watson Research Centre
More informationAttention-Based Temporal Weighted Convolutional Neural Network for Action Recognition
Attention-Based Temporal Weighted Convolutional Neural Network for Action Recognition Jinliang Zang 1, Le Wang 1(&), Ziyi Liu 1, Qilin Zhang 2, Gang Hua 3, and Nanning Zheng 1 1 Xi an Jiaotong University,
More informationarxiv:submit/ [cs.cv] 16 Jun 2017
The Monkeytyping Solution to the YouTube-8M Video Understanding Challenge arxiv:submit/1922641 [cs.cv] 16 Jun 2017 He-Da Wang whd.thu@gmail.com Ji Wu Teng Zhang zhangteng1887@gmail.com wuji ee@mail.tsinghua.edu.cn
More informationarxiv: v1 [cs.cv] 29 Sep 2016
arxiv:1609.09545v1 [cs.cv] 29 Sep 2016 Two-stage Convolutional Part Heatmap Regression for the 1st 3D Face Alignment in the Wild (3DFAW) Challenge Adrian Bulat and Georgios Tzimiropoulos Computer Vision
More informationarxiv: v1 [cs.cv] 4 Dec 2018
Classifying Collisions with Spatio-Temporal Action Graph Networks Roei Herzig 1, Elad Levi 2, Huijuan Xu 3, Eli Brosh 2, Amir Globerson 1, Trevor Darrell 2,3 1 Tel Aviv Univeristy, 2 Nexar, 3 UC Berkeley
More informationarxiv: v1 [cs.cv] 13 Aug 2017
Lattice Long Short-Term Memory for Human Action Recognition Lin Sun 1,2, Kui Jia 3, Kevin Chen 2, Dit Yan Yeung 1, Bertram E. Shi 1, and Silvio Savarese 2 arxiv:1708.03958v1 [cs.cv] 13 Aug 2017 1 The Hong
More informationOptical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition Shuyang Sun 1,2, Zhanghui Kuang 2, Lu Sheng 3, Wanli Ouyang 1, Wei Zhang 2 1 The University of Sydney 2
More informationVideo Gesture Recognition with RGB-D-S Data Based on 3D Convolutional Networks
Video Gesture Recognition with RGB-D-S Data Based on 3D Convolutional Networks August 16, 2016 1 Team details Team name FLiXT Team leader name Yunan Li Team leader address, phone number and email address:
More informationVGR-Net: A View Invariant Gait Recognition Network
VGR-Net: A View Invariant Gait Recognition Network Human gait has many advantages over the conventional biometric traits (like fingerprint, ear, iris etc.) such as its non-invasive nature and comprehensible
More informationCEA LIST s participation to the Scalable Concept Image Annotation task of ImageCLEF 2015
CEA LIST s participation to the Scalable Concept Image Annotation task of ImageCLEF 2015 Etienne Gadeski, Hervé Le Borgne, and Adrian Popescu CEA, LIST, Laboratory of Vision and Content Engineering, France
More informationTwo-Stream Convolutional Networks for Action Recognition in Videos
Two-Stream Convolutional Networks for Action Recognition in Videos Karen Simonyan Andrew Zisserman Cemil Zalluhoğlu Introduction Aim Extend deep Convolution Networks to action recognition in video. Motivation
More informationPerson Action Recognition/Detection
Person Action Recognition/Detection Fabrício Ceschin Visão Computacional Prof. David Menotti Departamento de Informática - Universidade Federal do Paraná 1 In object recognition: is there a chair in the
More informationREGION AVERAGE POOLING FOR CONTEXT-AWARE OBJECT DETECTION
REGION AVERAGE POOLING FOR CONTEXT-AWARE OBJECT DETECTION Kingsley Kuan 1, Gaurav Manek 1, Jie Lin 1, Yuan Fang 1, Vijay Chandrasekhar 1,2 Institute for Infocomm Research, A*STAR, Singapore 1 Nanyang Technological
More informationarxiv: v3 [cs.cv] 8 May 2015
Exploiting Image-trained CNN Architectures for Unconstrained Video Classification arxiv:503.0444v3 [cs.cv] 8 May 205 Shengxin Zha Northwestern University Evanston IL USA szha@u.northwestern.edu Abstract
More informationTemporal Action Detection with Structured Segment Networks
Temporal Action Detection with Structured Segment Networks Yue Zhao 1, Yuanjun Xiong 1, Limin Wang 2, Zhirong Wu 1, Xiaoou Tang 1, and Dahua Lin 1 1 Department of Information Engineering, The Chinese University
More informationarxiv: v1 [cs.cv] 6 Jul 2016
arxiv:1607.01794v1 [cs.cv] 6 Jul 2016 VideoLSTM Convolves, Attends and Flows for Action Recognition Zhenyang Li, Efstratios Gavves, Mihir Jain, and Cees G. M. Snoek QUVA Lab, University of Amsterdam Abstract.
More informationLSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University
LSTM and its variants for visual recognition Xiaodan Liang xdliang328@gmail.com Sun Yat-sen University Outline Context Modelling with CNN LSTM and its Variants LSTM Architecture Variants Application in
More informationAction recognition in videos
Action recognition in videos Cordelia Schmid INRIA Grenoble Joint work with V. Ferrari, A. Gaidon, Z. Harchaoui, A. Klaeser, A. Prest, H. Wang Action recognition - goal Short actions, i.e. drinking, sit
More informationGraph CNNs with Motif and Variable Temporal Block for Skeleton-based Action Recognition
Graph CNNs with Motif and Variable Temporal Block for Skeleton-based Action Recognition Abstract Hierarchical structure and different semantic roles of joints in human skeleton convey important information
More informationarxiv: v1 [cs.cv] 13 Dec 2018
Nrityantar: Pose oblivious Indian classical dance sequence classification system Vinay Kaushik Dept. of Electrical Engineering IIT Delhi eez158117@ee.iitd.ac.in Prerana Mukherjee Dept. of Electrical Engineering
More informationGAIT RECOGNITION BASED ON CONVOLUTIONAL NEURAL NETWORKS
GAIT RECOGNITION BASED ON CONVOLUTIONAL NEURAL NETWORKS A. Sokolova a, A.Konushin a, b a National Research University Higher School of Economics, Moscow, Russia - ale4kasokolova@gmail.com, akonushin@hse.ru
More informationDeep Learning in Visual Recognition. Thanks Da Zhang for the slides
Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object
More informationOn-line Television Stream Classification by Genre
Baltic J. Modern Computing, Vol. 6 (2018), No. 3, 235-246 https://doi.org/10.22364/bjmc.2018.6.3.02 On-line Television Stream Classification by Genre Karlis Martins BRIEDIS, Karlis FREIVALDS Faculty of
More informationKnow your data - many types of networks
Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for
More informationDynamic Image Networks for Action Recognition
Dynamic Image Networks for Action Recognition Hakan Bilen Basura Fernando Efstratios Gavves Andrea Vedaldi Stephen Gould University of Oxford The Australian National University QUVA Lab, University of
More informationDeep Tracking: Biologically Inspired Tracking with Deep Convolutional Networks
Deep Tracking: Biologically Inspired Tracking with Deep Convolutional Networks Si Chen The George Washington University sichen@gwmail.gwu.edu Meera Hahn Emory University mhahn7@emory.edu Mentor: Afshin
More informationHuman Motion Reconstruction from Action Video Data Using a 3-Layer-LSTM
Human Motion Reconstruction from Action Video Data Using a 3-Layer-LSTM Jihee Hwang Stanford Computer Science jiheeh@stanford.edu Danish Shabbir Stanford Electrical Engineering danishs@stanford.edu Abstract
More informationUTS submission to Google YouTube-8M Challenge 2017
UTS submission to Google YouTube-8M Challenge 2017 Linchao Zhu Yanbin Liu Yi Yang University of Technology Sydney {zhulinchao7, csyanbin, yee.i.yang}@gmail.com Abstract In this paper, we present our solution
More informationFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun Presented by Tushar Bansal Objective 1. Get bounding box for all objects
More informationarxiv: v2 [cs.cv] 13 Apr 2015
Beyond Short Snippets: Deep Networks for Video Classification arxiv:1503.08909v2 [cs.cv] 13 Apr 2015 Joe Yue-Hei Ng 1 yhng@umiacs.umd.edu Oriol Vinyals 3 vinyals@google.com Matthew Hausknecht 2 mhauskn@cs.utexas.edu
More information3D CONVOLUTIONAL NEURAL NETWORK WITH MULTI-MODEL FRAMEWORK FOR ACTION RECOGNITION
3D CONVOLUTIONAL NEURAL NETWORK WITH MULTI-MODEL FRAMEWORK FOR ACTION RECOGNITION Longlong Jing 1, Yuancheng Ye 1, Xiaodong Yang 3, Yingli Tian 1,2 1 The Graduate Center, 2 The City College, City University
More informationNeural Networks with Input Specified Thresholds
Neural Networks with Input Specified Thresholds Fei Liu Stanford University liufei@stanford.edu Junyang Qian Stanford University junyangq@stanford.edu Abstract In this project report, we propose a method
More informationEncoder-Decoder Networks for Semantic Segmentation. Sachin Mehta
Encoder-Decoder Networks for Semantic Segmentation Sachin Mehta Outline > Overview of Semantic Segmentation > Encoder-Decoder Networks > Results What is Semantic Segmentation? Input: RGB Image Output:
More informationarxiv: v1 [cs.cv] 2 May 2015
Dense Optical Flow Prediction from a Static Image Jacob Walker, Abhinav Gupta, and Martial Hebert Robotics Institute, Carnegie Mellon University {jcwalker, abhinavg, hebert}@cs.cmu.edu arxiv:1505.00295v1
More informationImage Captioning with Object Detection and Localization
Image Captioning with Object Detection and Localization Zhongliang Yang, Yu-Jin Zhang, Sadaqat ur Rehman, Yongfeng Huang, Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
More informationTALL: Temporal Activity Localization via Language Query
TALL: Temporal Activity Localization via Language Query Jiyang Gao 1 Chen Sun 2 Zhenheng Yang 1 Ram Nevatia 1 1 University of Southern California 2 Google Research {jiyangga, zhenheny, nevatia}@usc.edu,
More informationHUMAN ACTION RECOGNITION
HUMAN ACTION RECOGNITION Human Action Recognition 1. Hand crafted feature + Shallow classifier 2. Human localization + (Hand crafted features) + 3D CNN Input is a small chunk of video 3. 3D CNN Input is
More informationA FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS. Kuan-Chuan Peng and Tsuhan Chen
A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS Kuan-Chuan Peng and Tsuhan Chen School of Electrical and Computer Engineering, Cornell University, Ithaca, NY
More informationFeature-Fused SSD: Fast Detection for Small Objects
Feature-Fused SSD: Fast Detection for Small Objects Guimei Cao, Xuemei Xie, Wenzhe Yang, Quan Liao, Guangming Shi, Jinjian Wu School of Electronic Engineering, Xidian University, China xmxie@mail.xidian.edu.cn
More informationReal-time Action Recognition with Enhanced Motion Vector CNNs
Real-time Action Recognition with Enhanced Motion Vector CNNs Bowen Zhang 1,2 Limin Wang 1,3 Zhe Wang 1 Yu Qiao 1 Hanli Wang 2 1 Shenzhen key lab of Comp. Vis. & Pat. Rec., Shenzhen Institutes of Advanced
More informationarxiv: v1 [cs.cv] 20 Dec 2016
End-to-End Pedestrian Collision Warning System based on a Convolutional Neural Network with Semantic Segmentation arxiv:1612.06558v1 [cs.cv] 20 Dec 2016 Heechul Jung heechul@dgist.ac.kr Min-Kook Choi mkchoi@dgist.ac.kr
More informationCultural Event Recognition by Subregion Classification with Convolutional Neural Network
Cultural Event Recognition by Subregion Classification with Convolutional Neural Network Sungheon Park and Nojun Kwak Graduate School of CST, Seoul National University Seoul, Korea {sungheonpark,nojunk}@snu.ac.kr
More informationACTION RECOGNITION WITH GRADIENT BOUNDARY CONVOLUTIONAL NETWORK
ACTION RECOGNITION WITH GRADIENT BOUNDARY CONVOLUTIONAL NETWORK Huafeng Chen 1,2, Jun Chen 1,2, Chen Chen 3, Ruimin Hu 1,2 1 Research Institute of Shenzhen, Wuhan University, Shenzhen, China 2 National
More informationBody Joint guided 3D Deep Convolutional Descriptors for Action Recognition
1 Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition arxiv:1704.07160v2 [cs.cv] 25 Apr 2017 Congqi Cao, Yifan Zhang, Member, IEEE, Chunjie Zhang, Member, IEEE, and Hanqing Lu, Senior
More information