Evaluation of Triple-Stream Convolutional Networks for Action Recognition

Transcription

1 Evaluation of Triple-Stream Convolutional Networks for Action Recognition Dichao Liu, Yu Wang and Jien Kato Graduate School of Informatics Nagoya University Nagoya, Japan {liu, ywang, jien} (at) mv.ss.is.nagoya-u.ac.jp Abstract Recently, Two-Stream Convolutional Network has achieved remarkable performance. Especially, by capturing appearance and motion information, spatial-temporal two-stream networks bring noticeable improvement. On the other hand, dynamic image, which is a powerful representation for videos, has also been confirmed to provide complimentary information to spatial appearance. Inspired by these works, we proposed Triple- Stream Convolutional Networks by fusing a third network stream whose input is dynamic image. In this paper, we implement the proposed Triple-Stream Convolutional Networks and evaluated them in two aspects: (a) how the overall end-to-end classification performance can be benefited by adding the dynamic stream; (b) which way is efficient to use the trained Triple-Stream Convolutional Networks in classification. Our evaluation shows improvements over both single networks (spatial and temporal) and Fused Spatial-temporal Two-Stream Network. I. INTRODUCTION Action recognition in videos is the basic step of many promising applications. It is usually approached by two kinds of methods. One is shallow method, which uses the statistics of hand-craft local features to capture the spatial-temporal characteristics of actions [1], [2], [3]. The other is deep method, which uses deep models to learn the representation of actions in an end-to-end style [4], [5], [6], [3], [7], [8], [9]. While shallow method is good, more and more recent works show the power of deep methods. Two-stream convolutional network [10] is a very famous one of them. It trains two networks, namely spatial network and temporal network. Spatial network is fed with static frames to get the spatial clues on object appearance. Temporal network is fed with optical flows to get the temporal clues on object motion. These two kinds of clues are combined by fusing the two networks [10]. This work is very successful because this model is able to recognize both the objects and the motion. Many later researches are developed based on two-stream network. Some of the researches develop video representations based on it [11], [12]. Some other researches focus on the end-toend improvement over it [5], [9]. Especially in [5], spatial and temporal networks are fused and trained together, achieving state-of-art performance. Inspired by above works on two-stream networks, we propose the triple-stream network architecture. Triple-stream network is an extension of two-stream network, in which a third network is used to process the dynamic image inputs. Dynamic image [13] is a compact yet powerful representation for videos based on Rank Pooling [14], [15]. In an earlier work [16], it has been confirmed that dynamic image is efficient in end-to-end learning on action categories. As shown in Figure 1, dynamic images intuitively look like panning or motion blur, containing temporal evolution information in a static image. Traditional two-stream convolutional networks use static frames and optical flows (as shown in Figure 1) as inputs. We hypothesize that, as an intermediary feature containing both spatial and temporal information, dynamic image would help the model better associate spatial and temporal information. In this paper, our main purpose is to evaluate how classification performance can be improved by adding dynamic image network and to find out how to efficiently make use of the triple-stream architecture. Fig. 1: Example of a static frame (a), a dynamic image (b) and a stack of optical flows (c). We regard dynamic image as an intermediary feature between appearance and motion and thus hypothesize it can help improve the performance. II. EVALUATION METHODS We use three fusion methods to construct, train and utilize fused networks, namely Convolutional Fusion, Score Fusion and Feature Fusion, as shown in Figure 2. We first evaluate the performance improvement brought by fusing dynamic image stream into single spatial and temporal networks. We separately train single spatial, temporal and dynamic image networks. Then we fuse the dynamic image network respectively into the single spatial and temporal networks by the three methods. After that we implement triple-stream networks respectively by the three methods to evaluate the triple-stream networks performance /17/$ IEEE 513

2 Fig. 2: Illustration of process of convolutional fusion (a), score fusion(b) and feature fusion(c). After the fusion, we keep all the network towers and use all of them for training and prediction. It is because: (a) When training networks, reserving all towers generally can help the whole model optimize towards the target that all streams performance will be benefited; (b) When using networks for classification, the deep features or prediction scores from different towers always complement each other. A. Fusion methods Convolutional fusion (CF) [5], [17] fuses different networks together and and then trains and utilizes fused networks together. CF has been proved to be effective for fusing networks of RGB frame inputs and optical flow inputs [5]. The advantage of this method is that it can exploit activations correspondence of the same pixel position. By using this method, [5] is able to fuse appearance and motion feature maps correspondingly and thus achieved remarkable improvement. However, as we observe, when more networks are fused together, shortcomings may also emerge, such as it tends to be more difficult to optimize the fused networks. We use CF to fuse the feature maps of ReLU5 layer from different networks. As shown in Figure 2(a), CF concatenates the feature maps from different networks together and then convolves the activations of concatenated feature maps. Similar to the convolutional fusion layer for two networks constructed in [5], we constructed convolutional fusion layer for more networks. As illustrated in equation (1), x 1, x 2,...x n are the D-channel feature maps. The operator cat means to concatenate. Concatenated feature maps are then fused by filter f R 1 1 nd D and the bias b R D. During training, f can fuse the concatenated feature maps correspondingly with dimensionality reduction. The fusion is weighted and the weights can be adjusted by minimizing joint loss when the whole networks are being trained. y = cat(x 1,x 2,...x n ) f + b (1) Score fusion (SF), as shown in Figure 2(b), firstly trains the networks separately. Then all the parameters are fixed and the networks are fused at prediction layer by average fusion. In our case, we fuse the activations of fc8 layer to be the final prediction scores. The advantage of this method is that separately trained networks are less complicated thus easier to optimize. However the disadvantage is a lack of correspondence of feature maps from different network streams. Feature fusion (FF) extracts CNN activations and then fuses the activations from different streams to be video representations. As shown in Figure 2(c), we extract the ReLU7 activations of each stream as respective video representations of each stream. Then the respective representations are aggregated via average pooling and then implemented with l2 normalization. After that, data whitening is implemented on the averaged scores to reduce dimensional dependency. Finally, we implement l2 normalization again and got our deep-learned representation for each video. B. Network fusion To evaluate how the recognition performance can be benefited from fusing dynamic image stream, we use above fusion methods to fuse dynamic image into two types of networks: one is single spatial or temporal networks and the other is spatial-temporal two-stream network [5]. Firstly, we fuse dynamic image stream with single networks and compare the performance. As shown in in Figure 3, we separately train single spatial, temporal and dynamic image networks. Then we respectively fuse dynamic image network into single spatial and temporal networks using CF, SF and FF. We evaluate whether and how the performance will be benefited by adding dynamic image stream to single networks respectively by the three fusion methods. Then, as shown in in Figure 4, we separately train a single dynamic image network and a fused spatial-temporal twostream network rather than training three single networks separately. We construct the triple-stream networks by fusing dynamic image stream into the spatial-temporal two-stream network respectively by the three fusion methods. For CF, we fuse the dynamic image stream into spatial-temporal twostream network at the same fusion layer where spatial and temporal streams are fused. For FF, we fuse the ReLu 7 activations 514

3 from all three streams together to be the video representation by the process mentioned in section II-A. However, for SF, since [13] shows dynamic images complementarity to spatial appearance to some extend, when averaging prediction scores, in order to reduce performance uncertainty of our evaluation, we average as equation (2) instead of just directly averaging the scores together. S = avg(avg(s sp,s dy ),S te ) (2) In equation (2), S is our final prediction score. S sp is the prediction score of spatial stream. S dy is the prediction score of dynamic image stream. S te is the prediction score of temporal stream. Operator avg means to average. Fig. 4: Triple-stream networks constructed by fusing dynamic image stream using CF (a), SF (b) or FF (c). Same as Fig. 3, conv here also represents layer groups. For triple-stream CF and FF, we implement by similarly way of two-stream CF and FF. For triple-stream SF, we first average the prediction scores from spatial stream and dynamic image stream. Then we average the averaged scores and the prediction scores from temporal stream. Fig. 3: Two-stream networks constructed by fusing dynamic image stream using CF (a), SF (b) or FF (c). In this figure, conv actually represents layer groups. For example, conv5 represents the layers conv5 1...conv5 3 in VGG16. III. EXPERIMENT AND EVALUATION We evaluate on the UCF101 dataset [18], which is a popular dataset consisting video clips in 101 categories. We extract RGB frames and compute optical flows from all the frames for each video. Then multiple dynamic images are generated from each video for every L dy = 10 frames. When training, dynamic image stream is fed with dynamic images while spatial and temporal stream are respectively fed with RGB frames and optical flow stacks of L te = 10 frames. In our experiment, we first evaluate in the 1st split of UCF101 to see the performance, and then the methods of higher performance will be evaluated in all three splits for testing performance consistency. A. Implementation details Details of network training: All the networks in this paper are based on the very deep VGG-16 model [19] which has totally 16 layers (13 convolutional and 3 fully-connected layers). When training or fine-tuning networks, the batch size is set as 96. To save memory, instead of computing the gradients of a whole batch in one go, we split every batch into smaller sub-batches, and accumulating gradients by processing them sequentially. We do data augmentation (randomly cropping and flipping) for the inputs of all networks. Like the implementation in [5], the learning rate of every network starts from 10 3 and then gets reduced to its 10% when training accuracy saturation happens, until it reaches to We apply dropout for first two fully-connected layers of every stream. When training single networks, dropout ratios of both spatial network and dynamic image network are set as 0.85 while the dropout ratio of temporal network are set as 0.9. When training different two-stream or triple-stream networks, all the dropout ratios are set as For validation, we sample T = 5 frames (T static frames, T stacks of optical flows and T dynamic images) with their horizontal flips without cropping for each video. Details of FF construction: To construct deep-learned representations, we also sample T = 5 frames with their horizontal flips without cropping for each video. Then these frames are fed into trained networks and the 4096-D features of ReLU7 layer are extracted by the process in section II-A. B. Performance of fusing dynamic image network into single networks First, we evaluate the effects of fusing dynamic image network with single spatial and temporal networks. The results 515

4 TABLE I: Effect evaluation of fusing dynamic image network to single spatial or temporal networks Method single network CF SF FF or Deep feature of single-stream network dynamic image 79.36% % spatial 82.13% % spatial+dynamic image % 85.62% 84.46% temporal 85.04% % temporal+dynamic image % 88.30% 89.32% are shown as in Table I. It can be observed that, when spatial and dynamic image network are fused by SF, the performance was improved remarkably. The similar improvement has also been seen in the implementation of [18]. Besides, we show that, by fusing temporal and dynamic image network using SF, performance can be also improved over single temporal network. FF also performs much better than single networks. Note that our deep-learned representation of single networks performs similarly to its end-to-end implementation. Thus it can be asserted that the performance improvement is contributed by the addition of dynamic image network rather than only the representation process itself. However, when networks are fused by CF, performance decreases for the both input combinations. Our results show that dynamic image network is not suitable for being fused and trained together with spatial network or temporal network by CF. We are going to discuss about possible reason in section III-D. C. Performance of triple-stream networks Then, we compare the performance of triple-stream networks with the performance of spatial-temporal two-stream CF [5]. As shown in Table II, both triple-stream SF and triplestream FF performs better than spatial-temporal two-stream fusion. However, triple-stream network CF performs a little worse than spatial-temporal two-steam CF. We are going to discuss about possible reason in section III-D. TABLE II: Comparison between spatial-temporal two-stream CF and triple-stream networks in UCF101 split1 Method Accuracy Spatial-temporal two-stream CF 90.70% Triple-stream CF 90.30% Triple-stream SF 91.03% Triple-stream FF 90.87% Finally, we evaluate triple-stream networks of higher performance (SF and FF) in all three splits of UCF 101. As shown in Table III, both of the two methods perform better than twostream CF in all three splits. It shows that the two methods can gain consistent improvement over two-stream CF. D. Discussion for unsatisfying performance of fusing dynamic image network by CF We observe that, the networks would be much harder to optimize when dynamic image network is fused by CF. For TABLE III: Comparison between spatial-temporal two-stream CF and triple-stream networks in three UCF101 splits Method split 1 split 2 split 3 Average Spatial-temporal two-stream CF 90.70% 90.80% 91.59% 91.03% Triple-stream SF 91.03% 91.67% 92.53% 91.74% Triple-stream FF 90.87% 91.38% 92.67% 91.64% example, Table IV shows the training statuses of spatialtemporal two-stream CF and triple-stream CF when the statuses saturated. Both training error and training loss in spatial and temporal stream of the latter were much higher than those of the former. In terms of the possible reason for optimizing difficulty, as we observe, it lies in the redundancy of information provided by dynamic image and spatial stream, dynamic image and temporal stream. To confirm this, we randomly select 200 video frames and their corresponding dynamic images, stacks of optical flows respectively from 200 different videos. Then we feed them into triple-stream network and extract the feature maps of ReLU5 layer, which are the feature maps we concatenate for CF evaluation. We then convert the dimensional feature maps into dimensional vectors. We refer such vectors as R5Vs. We compute the average correlation coefficient among R5Vs from different streams, the results are shown in Table V. Compared with correlation between the R5Vs from spatial and temporal stream, the correlation between the R5Vs from spatial and dynamic image stream, temporal and dynamic image stream are much higher. We also try to further illustrate this situation in Figure 5 by projecting R5Vs into a two-dimensional surface. As is known, high redundancy and strong correlations in data always result in difficulty of optimization and thus cause the unsatisfying performance of many machine learning algorithms [20], [21], [22]. R5Vs are actually the inputs of following fully-connected layers. The redundancy of R5Vs can cause bad influence on the classification ability of fullyconnected layers. On the contrary, in SF, the prediction scores are already the results from optimization so the bad influence of redundancy is limited. Also, in FF, because data whitening is implemented, even though we keep the same dimension of datas after whitening, the correlation among datas is significantly reduced. Thus, no matter FF or SF can be benefited from fusing dynamic image stream. Our future goal is to find ways to reduce the correlation and redundancy in the concatenated feature maps after dynamic image stream is fused in end-to- 516

5 end manner. TABLE IV: Comparison of final training statuses between spatial-temporal two-stream CF and triple-stream CF Networks Error Loss Spatial-temporal Spatial stream two-stream CF Temporal stream Triple-stream CF Spatial stream Temporal stream TABLE V: Average correlation coefficient among R5Vs from different streams Spatial-temporal Spatial-dynamic Temporal-dynamic Fig. 5: Illustration of two-dimensional points projected from R5Vs of spatial (blue rhombus), temporal (red square) and dynamic image (green triangle) streams. Even though some blue rhombus mix with red squares in position. Overall, it is clear that most of the blue rhombus assemble on the left while the red squares assemble on the right. Yet, compared with the tiny mixture of red square and blue rhombus, green triangles mix greatly with both two other figures. Thus it is obvious that there exists high redundancy between feature maps from dynamic image stream and two other streams. IV. CONCLUSION In our work, we first evaluate the effects of fusing dynamic image network to single spatial or temporal network. Except fusing related networks by CF, all methods gain noticeable improvement over single networks. Then we evaluate triplestream network. Except CF, the other two triple-stream fusion methods gain consistent improvement over two-stream fusion. Our results show that, by suitable fusion methods, the addition of dynamic image stream can help both single networks and fused spatial-temporal two-stream networks better recognize human actions in videos. Regarding the possible for unsatisfying performance of fusing dynamic image network by CF, we suppose it is the redundancy of information between dynamic image and spatial streams, dynamic image and temporal streams, which could make networks more difficult to optimize. We plan to further find out and prove the reason and come up with solutions to get better performance. ACKNOWLEDGEMENT This research is supported by the JSPS Grant-in-Aid for Challenging Exploratory Research (No.16K12460), the JSPS Grant-in-Aid for Young Scientists (B) (No.17K12714), the JST Center of Innovation Program and PhD Program Toryumon of Nagoya University. REFERENCES [1] H. Wang and C. Schmid, Action recognition with improved trajectories, in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp [2] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu, Dense trajectories and motion boundary descriptors for action recognition, International journal of computer vision, vol. 103, no. 1, pp , [3] L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi, Human action recognition using factorized spatio-temporal convolutional networks, in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp [4] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, Beyond short snippets: Deep networks for video classification, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp [5] C. Feichtenhofer, A. Pinz, and A. Zisserman, Convolutional two-stream network fusion for video action recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp [6] N. Ballas, L. Yao, C. Pal, and A. Courville, Delving deeper into convolutional networks for learning video representations, arxiv preprint arxiv: , [7] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, Joint fine-tuning in deep neural networks for facial expression recognition, in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp [8] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, Large-scale video classification with convolutional neural networks, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014, pp [9] C. Feichtenhofer, A. Pinz, and R. P. Wildes, Spatiotemporal residual networks for video action recognition, CoRR, vol. abs/ , [Online]. Available: [10] K. Simonyan and A. Zisserman, Two-stream convolutional networks for action recognition in videos, in Advances in neural information processing systems, 2014, pp [11] L. Wang, Y. Qiao, and X. Tang, Action recognition with trajectorypooled deep-convolutional descriptors, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp [12] R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. C. Russell, Actionvlad: Learning spatio-temporal aggregation for action classification, CoRR, vol. abs/ , [Online]. Available: [13] H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould, Dynamic image networks for action recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp [14] B. Fernando, E. Gavves, J. Oramas, A. Ghodrati, and T. Tuytelaars, Rank pooling for action recognition, IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 4, pp , [15] B. Fernando and S. Gould, Learning end-to-end video classification with rank-pooling, in International Conference on Machine Learning, 2016, pp [16] M. S. Aliakbarian, F. Saleh, B. Fernando, M. Salzmann, L. Petersson, and L. Andersson, Deep action-and context-aware sequence learning for activity recognition and anticipation, arxiv preprint arxiv: , [17] C. Xiong, L. Liu, X. Zhao, S. Yan, and T.-K. Kim, Convolutional fusion network for face verification in the wild, IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 3, pp ,

6 [18] K. Soomro, A. R. Zamir, and M. Shah, Ucf101: A dataset of 101 human actions classes from videos in the wild, arxiv preprint arxiv: , [19] K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arxiv preprint arxiv: , [20] L. Yu and H. Liu, Efficient feature selection via analysis of relevance and redundancy, Journal of machine learning research, vol. 5, no. Oct, pp , [21] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky, Sparse convolutional neural networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp [22] Z. Pan, A. G. Rust, and H. Bolouri, Image redundancy reduction for neural network classification using discrete cosine transforms, in Neural Networks, IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, vol. 3. IEEE, 2000, pp