Action Recognition Using Local SpatioTemporal Oriented Energy Features and Additive. Kernel SVMs

Transcription

1 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 Action Recognition Using Local SpatioTeporal Oriented Energy Features and Additive Kernel SVMs Jiangfeng Yang and Zheng Ma School of Counication and Inforation Engineering, University of Electronic Science and Technology of China, Chengdu, China Eail: 36933@qq.co, wallsonyang@63.co (STOE), which is derived fro the filter responses of orientation selective bandpass filters, as features has been successfully applied in the fields of video tracking and action recognition. Video sequences induce very different orientation patterns in iage spacetie depending on their contents. For instance, a textured, stationary object yields a uch different orientation signature than if the very sae object were undergoing translational otion. An efficient fraework for analyzing spatio-teporal (ST) inforation can be realized through the use of 3D, (x, y, t), oriented energies []. The aforeentioned energy is well-suited to for the feature representation in visual tracking and action recognition applications for three significant reasons. ()A rich description of the target is attained due to the fact that oriented energy encopasses both target appearance and dynaics. ()The oriented energy is robust to illuination changes. By construction, the proposed feature set provides invariance to both additive and ultiplicative iage intensity changes. (3)The energy can be coputed at ultiple scales, allowing for a ulti scale analysis of the target attributes. Finer scales provide inforation regarding otion of individual target parts (e.g., libs) and detailed spatial textures (e.g., facial expressions, clothing logos). In a copleentary fashion, coarser scales provide inforation regarding the overall target velocity and its gross shape []. Recently, local representations based on STIPs are drawing uch attention [3]-[5], aong which huan action recognition systes based on the bag of words (BoW) odel have achieved good results in any tasks. This would be due to the fact that the BoW odel has any advantages, such as being less sensitive to partial occlusions and clutter and avoiding soe preliinary steps, e.g. background subtraction and target tracking in holistic ethods. In our recognition schee, an action is considered as a congloeration of otion energies in different ST orientations. Consider that otion at a point is captured as a cobination of energies along different ST orientations at that point, when suitably decoposed. These decoposed otion energies are a low-level action representation and the basis of the action recognition Abstract Spatio-teporal oriented energy features have been proved to be an efficient feature for action recognition. It has satisfied perforance on ost of public databases. However, the oriented energy features were used as holistic action features for teplate atching in any literatures. In the paper, we proposed an action representation based on local spatio-teporal oriented energy features, and ultiple feature channels are built to convert the features to descriptors. Moreover, inspired by additive kernel Support Vector Machine can offer significant iproveents in accuracy on a wide variety of tasks while having the sae run-tie. We proposed action classifiers based on additive kernels and tested our syste on KTH huan action dataset for its perforance evaluation. The experiental result shows our syste outperfors ost of recent action classification systes. Index Ters action recognition, action representation, spatio-teporal oriented energy, additive kernels I. INTRODUCTION Autoatic recognition and categorization actions in video sequences is an active research topic in coputer vision and achine learning, and their applications can be found in any areas, including content-based video indexing, detecting activities and behaviors in surveillance videos, organizing digital video library according to specified actions, huan-coputer interfaces and robotics. The challenge is how to obtain robust action recognition and categorization under variable illuination, background changes, caera otion and zooing, viewpoint changes, and partial occlusion. Moreover, the intra-class variation is often very large and abiguity exists between actions. Feature representation as a fundaental part of action recognition will greatly influence the perforance of the recognition syste. Huan actions fro video data inherently contain both spatial and teporal inforation, which requires that descriptors of actions in video sequences accurately capture and robustly encode this kind of inforation. Spatio-teporal oriented energy Manuscript received October 9, 3; revised February, 4. Project nuber: National Nature Science Foundation of China (grant nuber 678). 4 Engineering and Technology Publishing doi:.7/ijeee

2 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 ethod [6]. Moreover, additive kernel (i.e., intersection kernels) Support Vector achines (SVMs) have becoe popular for real-tie applications as they enjoy both faster training and faster classification, with better classification accuracy and significantly less eory requireents than nonlinear kernels. Therefore, SVMs based on chi-squared, Jensen-Shannan (JS) and intersection kernel were eployed as action classifiers in the paper. We tested the proposed syste on KTH huan action dataset for perforance evaluation. To study cuboid size and codebook size ipacting upon recognition rate of action classifiers, the proposed schee was tested under four sizes of cuboid and codebook. In all cases, the teplate size of ST oriented filters is set to What is ore, to assess recognition accuracy and training tie consuption of classifiers based upon additive kernels, RBF and linear kernels were utilized in the paper. II. THE PROPOSED APPROACH Our action categorization syste is illustrated in Fig. The details of the proposed syste are explained as follows. Figure. Fraework of the proposed action classification syste. In dictionary learning stage, (a) Action otion inforation is decoposed into local STOE features along X, Y and T axis. (a) STOE features are transfored into descriptors. (a3) A feature channel consists of an energy orientation and a ST grid. A feature channel produces its own codebook. In training stage, (b) Input training action videos. (b) Action otion inforation is converted into local STOE features. (b3) Each action class is odeled by nine local histogras. (b4) Each action class is represented as a final histogra, which is built by concatenating its nine sub-histogras. of STIPs, which is iportant for action representation built upon BoW odel. In the following, we provide a brief review of the Dollar detector, which respectively eploys D Gaussian filter in the spatial direction and D Gabor filter in the teporal direction. The two separate filters can produce high response at points with significant ST intensity variations. The response function R of the input iage sequence I ( x, y, t ) has the for: A. Gaa Noralization and Frae Soothing Before extracting STIPs fro action video, Gaa noralization should be ipleented to enhance the local dynaic range of the iage in dark or shadowed regions, while copress it in bright regions and at highlights. The basic principle is that the intensity of the light reflected fro an object is the product of the incoing illuination (which is piecewise sooth for the ost part) and the local surface reflectance (which carries detailed object-level appearance inforation). In the paper, Gaa noralization is realized using Matlab coand J= iadjust(i), which aps the intensity values in grayscale iage I to new values in J. This increases the contrast of the output iage J. Moreover, to reduce the noise effect, Gaussian kernel with.5 as soothing filters is used to sooth each frae in action video. R ( I g hev ) ( I g hod ) where g ( x, y; ) is the D Gaussian soothing function, applied only along the spatial diensions, hev and hod are a quadrature pair of D Gabor filters applied teporally, which are defined as cos( t ) exp( t ) and, respectively. The sin( t ) exp( t ) paraeters and correspond to the spatial and teporal scales of the detector, respectively. It is well accepted that stable and reliable STIPs extracted fro action videos can offer ore helpful inforation to iprove recognition accuracy. In order to B. STIP Detector In the paper, the Dollar detector [4] is chosen to be STIP detector, since it generally produces a high nuber 4 Engineering and Technology Publishing () 5

3 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 extract such STIPs fro noisy videos, the response function should be calculated at ultiple spatial and teporal scales. Specifically, the response function R(, ) is defined as: M R(, ) R(, ),, () where R(, ) denotes the response value of STIP detector at paraeter values (, ). C. Local STOE Features Events in a video sequence will generate diverse structures in the ST doain. For instance, a textured, stationary object produces a uch different signature in iage space-tie than if the sae object were oving. One ethod of capturing the ST characteristics of a video sequence is through the use of oriented energy. The energy is derived using the filter responds of oriented selective bandpass filters when they are convolved with the ST cuboid produced by a video strea. In the paper, local STOE features are obtained by filtering using a set of Gaussian derivative filters and their corresponding Hilbert transfor filters, pointwise squaring and suation over each 3D cuboid that is associated with a detected STIP. We use the approach in [7] to obtain local STOE feature. Specifically, the first step consists of filtering a sall ST cuboid Cx ( i ) centering on a STIP p( x i ), by the directionally selective filters G and H at orientation vector θ (,, ), j{,, J},(,, ) denotes j j j j j j j direction cosines, J is the nuber of orientations. Next, the filters are taken in quadrature to eliinate phase sensitivity in the output of each filter. Local STOE energy E θ ( x ) j i in cuboid Cx ( i ) at orientation θ j can be coputed according to E ( x ) ( G C( x )) ( H C( x )) (3) θ j i, θ j i, θ j i where xi ( xi, yi, ti ), i {,, n} denotes STIP coordinates of n STIPs; is convolution; G denotes 3D steerable, separable filters based on Gaussian second derivative; H their corresponding Hilbert transfor. The ethod of constructing ST filters G and H in [7] was adopted in the paper. In our syste, otion inforation were decoposed into local STOE along X, Y and T axis using ST oriented filters, respectively. D. The Descriptor Once local STOE features were obtained, in this section, our goal is to convert the features into their descriptors. First of all, we define a support region as a cuboid containing STOE feature around a STIP, and the support region is divided into several cells by ST grids with ultiple teporal scales for coputing descriptor, in order to delineate the teporal variations of a local otion pattern and ebed ST structure inforation. Supposing a support region is divided into cells, where is spatial grid arrangeent, is teporal segent nuber. For each cell, the ean absolute difference (MAD, ), the second (variance, ), third (skewness, 3 ) central oents are coputed. The ean absolute difference (MAD) for cell c with size s=s s s3 is defined as MAD (,, ) ( ) s s s3 e u u u3 ean c (4) s u u u3 e u u u ean c k k s s s3 k ( (,, 3) ( )),,,3, (5) s u u u3 s s s3 ean( c) ( e( u, u, u )) (6) s u u u3 3 where e( u, u, u 3) denotes energy value at position ( u, u, u ). For a support region divided into cells, each 3 cell produces a colun vector,, 3 T. Hence, a 3 atrix M R is obtained to describe STOE feature inside the support region. Next, noralized atrix 3 M R is constructed by dividing each row of M by nor of the row. All row vectors of M are concatenated on top of each other to for a single d vector R, d 3. The vector is the descriptor of STOE feature inside the support region. In the proposed approach, three ST grids were established to partition a support region into, and 3 cells, respectively. The cobination of the three ST grids with three ST orientations results in 9 possible feature channels. Each feature channel decoposes otion inforation within a cuboid centering a STIP into local STOE feature, and partitions the support region into several cells. As each cell produces a 3-D vector, the diensionalities of descriptor corresponding to the above ST grids are, 4 and 36, respectively. Fig. illustrates that three ST grids transfor local STOE features into descriptors. Figure. A set of local STOE features is converted into three sets of descriptors using three ST grids. (a) A set of support regions. (b) ST grids with three teporal scales. (b) ST grid with size. (b) ST grid with size. (b3) ST grid with size 3. (c) Three sets of descriptors are built, using ST grids partitioning support regions. 4 Engineering and Technology Publishing 6

4 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 E. Codebooks The descriptors produced by a feature channel bring about a codebook by ipleenting K-eans clustering algorith and Euclidean distance as etric. Therefore, nine feature channels result in nine codebooks. The descriptors then are encoded by corresponding codebook to obtain local histogra. All local histogras are concatenated to for a final histogra for training and test the proposed syste. Hence, an action video sequence is represented as a final histogra vector. In our syste, all local histogras have sae length. F. Additive Kernel SVMs Discriinative classifiers based on Support Vector Machines (SVMs) and variants of boosted decision trees are two of the leading techniques used in vision tasks ranging fro object detection, ulticlass object recognition to texture classification. Classifiers based on boosted decision trees such as [8], have faster classification speed, but are significantly slower to train. Furtherore, the coplexity of training can grow exponentially with the nuber of classes [9]. On the other hand, given the right feature space, SVMs can be ore efficient during training. Part of the appeal of SVMs is that, nonlinear decision boundaries can be learnt using the kernel trick. However, the run-tie coplexity of a nonlinear SVM classifier can be significantly higher than a linear SVM. Thus, linear kernel SVMs have becoe popular for real-tie applications as they enjoy both faster training and faster classification, with significantly less eory requireents than non-linear kernels. Recently, additive kernel SVMs (AKSVMs) have been broadly used in any of the current ost successful object detection and recognition algoriths efficient, as well as real-tie applications. The class of kernels includes the intersection kernel K (, ) in xy, chi-squared kernel K ( xy, ), and JS kernel K ( xy, ). They are defined as following, K JS K in i n JS ( xy, ) in xi, yi (8) n K ( xy, ) ( x y ) ( x y ) (9) i i i i i x x y y x y ( x, y) ( )log ( )log () n i i i i i i i xi yi In addition of the additive kernels provided, RBF kernel KRBF ( xy, ) and linear kernel Klinear ( xy, ) are eployed as perforance benchark in our experient. K ( x, y) exp( x y ), () RBF III. Klinear T ( x, y) x. y () EXPERIMENTAL RESULT A. Experiental Setting We test our algorith on the KTH huan otion dataset. KTH huan otion dataset is a video sequence dataset of huan actions. Each video has only one action. The dataset contains six types of huan actions (walking, jogging, running, boxing, hand waving and hand clapping) perfored several periods by 5 subjects in different scenarios of outdoor and indoor environent with scale change. It contains 6 short sequences. In the preprocessing stage, contrast noralization is used to reduce the influence of illuination changes. STIPs are extracted fro action video using Dollar STIP detector with ultiple teporal and spatial scales. To extract stable, reliable STIPs,, are set to {(, ).,.,.3;.,.3,.4,.5}. STIP detector respond is obtained by suing over the individual responds. We build codebooks fro all videos of each action fro two subjects. We perfor leave-oneout (LOO) to test the efficiency of our approach in recognition. Specifically, for each LOO run, the videos of 4 subjects are used for training a odel, and the videos of the reained subject are used for testing the odel, and coputing a confusion atrix for evaluation. The results are reported as the average confusion atrix of 5runs. Accuracy rate of the recognition syste based on BoW odel is sensitive to the size of cuboid. Too big cuboid can obtain uch irrelevant inforation to action otion, such that the discriinative power of individual cuboid is daaged; whereas, too sall size cuboid enclose insufficient otion inforation for categorization, and hence the inforation provided is highly unreliable. We test our syste in the case of four cuboid sizes: 9(pixels) 9(pixels) 5(pixels), 9 9 9, and Moreover, coputational efficiency and accuracy are discussed in the section. B. Discussion Table I shows classification accuracy of SVMs based on five kernels in the case of four cuboid sizes, respectively. Recognition accuracy. We copared the average recognition accuracy of the five kernel SVMs on KTH dataset. It can be seen that recognition accuracy of AKSVMs is higher than that of others under the sae configuration. The best recognition rates of chi-squared, JS, and intersection kernel SVMs achieve 94.%, 93.9%, and 93.6%, respectively, which outperfor or approach the recent action recognition systes. Meanwhile, recognition accuracy of SVMs based on RBF and linear kernels just reach 9.5% and 9.7%, respectively. Table II shows recognition rate coparison of our ethod and other recent recognition systes. It can be seen that our systes achieve better perforance. The partial reason why AKSVM classifiers perfor better than that based on RBF, linear kernels lies in that: ST grids, which partition support region for coputing descriptor, essentially involve overlaps between the, although the ST grids are in general expected to be copleentary to each other. The best perforance of a 4 Engineering and Technology Publishing 7

5 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 particular action category generally entails only a linear cobination of the ST grids. As a result, there is no doubt that the redundant inforation exist in the resulting local histogras. It is concluded that the AKSVM classifiers efficiently exploit the redundant inforation to iprove their recognition rate; RBF and linear kernel classifiers fail to efficiently do it. TABLE I. CLASSIFICATION ACCURACY ON THE KTH DATASET AT VARIOUS CUBOID SIZES AND CODEBOOK SIZES. Cuboid size: Chi-sq JS Inters RBF Linear Cuboid size: Chi-sq JS Inters RBF Linear Cuboid size: Chi-sq JS Inters RBF Linear Cuboid size: Chi-sq JS Inters RBF Linear The size of cuboid is another reason ipacting on recognition rate. Too larger or too sall cuboid size could bring about negative effect to recognition rate as recognition rate. Too larger or too sall cuboid size could bring about negative effect to recognition rate discussed above. Table III and Table IV show confusion atrixes of AKSVMs for the KTH dataset under various cuboid sizes. The confusion atrixes show the largest confusion occurs between boxing and hand clapping, walking and jogging, and between jogging and running. Several reasons should be responsible for such confusion. Firstly, the decoposed otion energy of the actions ainly distributes along certain orientation, such as significant aount of discoposed otion energy of boxing and hand clapping distributes along X axis. Hence, the inforation obtained about action otion direction is insufficient for distinguishing the. Secondly, the speed difference of siilar action class is so sall that distinguishing the is difficulty, especially between walking and jogging, and between jogging and running. TABLE III. CONFUSION MATRIXES OF AKSVMS FOR THE KTH DATASET. CHI-SQUARED (TOP LEFT), JS (TOP RIGHT), AND INTERSECTION KERNELS (BOTTOM). ROWS ARE GROUND TRUTH, COLUMNS ARE MODEL RESULTS. BOXING(S), HAND-CLAPPING(S), HAND-WAVING(), JOGGING(S4), WALKING(S5), RUNNING (S6). each local histogra with 7D and cuboid size Average recognitions are 93.3%, 93.6% and 9.5% for chi-squared, JS, and intersection kernels, respectively. S S S4 S5 S6 S 96,96 3,3,3, 96 3 S 3,3 96,96, 3 96,, 96,96 96 S4 97,96 3, S5,4 88,86, S6, 88,4 87,85 87 TABLE II. COMPARISON WITH PREVIOUS WORK ON THE KTH DATASET. Method Accuracy [] 9.8% [9] 93.% [] 93.9% [] 94.5% [3] 94.% [4] 94.5% [5] 93.9% [6] 93.6% Our ethod (chi-sq. kernel) 94.% Our ethod (JS kernel) 93.9% Our ethod (Inters. Kernel) 93.6% each local histogra with 5D and cuboid size Average recognitions are 93.7%, 93.9% and 93.6% for chi-squared, JS and intersection kernel, respectively. S S S4 S5 S6 S 95,96 4,3,, 96 3 S 3,3 96,96, 3 96,, 97,97 96 S4 99,98, 97 3 S5 3,4 89,87 5,9 S6, 88 3,3 86, Engineering and Technology Publishing 8

6 International Journal of Electronics and Electrical Engineering Vol., No., June, 4 TABLE IV. CONFUSION MATRIXES OF AKSVMS FOR THE KTH DATASET. CHI-SQUARED (TOP LEFT), JS (TOP RIGHT), AND INTERSECTION KERNELS (BOTTOM). ROWS ARE GROUND TRUTH, COLUMNS ARE MODEL RESULTS. BOXING(S), HAND-CLAPPING(S), HAND-WAVING(), JOGGING(S4), WALKING(S5), RUNNING (S6). each local histogra with 7D and cuboid size Average recognitions are 94.%, 9.5% and 93.% for chi-squared, JS, and intersection kernels, respectively. S S S4 S5 S6 S 97,96,3, 97 S 3,3 96,96, 3 96,, 97,97 96 S4 98,97, S5 3,4 89,88 8,8 S6, 88,4 88,85 87 each local histogra with 5D and cuboid size Average recognitions are 9.5%, 9.8% and 9.5% for chi-squared, JS, intersection kernels, respectively. S S S4 S5 S6 S 96,96 3,3, 96 3 S 3,3 95,95, 3 95,, 97,97 96 S4 97,95 3, S5 5,5 86,8 9,3 S6 IV. 4, CONCLUSION 84 5,5 4 84,84 85 In the paper, we presented an action recognition syste based on local STOE feature and AKSVMs. Using ST oriented filters, otion inforation is decoposed into STOE feature. Then, ultiple feature channels are used to convert STOE features to descriptors. Finally, AKSVMs are eployed as action classifiers. The experiental result certifies that the proposed syste achieves better recognition accuracy. ACKNOWLEDGMENT This work is supported by National Nature Science Foundation of China (grant nuber 678). REFERENCES [] E. H. Adelson and J. R. Bergen, Spatioteporal energy odels for the perception of otion, Journal of the Optical Society of Aerica, vol., no., pp , 985. [] K. J. Cannons, J. M. Gryn, and R. P. Wildes, Visual tracking using a pixelwise spatioteporal oriented energy representation, in Proc. European Conference on Coputer Vision,, pp [3] I. Laptev and T. Lindeberg, Local descriptors for spatioteporal recognition, in Spatial Coherence for Visual Motion Analysis, 6, pp [4] P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, Behavior recognition via sparse spatio-teporal features, in Visual Surveillance and Perforance Evaluation of Tracking and Surveillance, 5, pp [5] G. Willes, T. Tuytelaars, and L. V. Gool, An efficient dense and scale-invariant spatio-teporal interest point detector, in Coputer Vision-ECCV, 8, pp [6] K. G. Derpanis, M. Sizintsev, and K. Cannons, Efficient action spotting based on a spacetie oriented structure representation, in Proc. the IEEE Coputer Vision and Pattern Recognition,, pp [7] K. G. Derpanis and J. M. Gryn, "Three-diensional nth derivative of Gaussian separable steerable filters," in Proc. the International Conference on Iage Processing, 5. [8] A. Torralba, K. Murphy, and W. Freean, Sharing features: Efficient boosting procedures for ulticlass object detection, in Coputer Vision and Pattern Recognition, 4. [9] M. Bregonzio, S. Gong, and T. Xiang, "Recognising action as clouds of space-tie interest points," in Proc. the IEEE Coputer Vision and Pattern Recognition, 9, pp [] I. Laptev, M. Marszalek, C. Schid, and B. Rozenfeld, Learning realistic huan actions fro ovies, in Proc. the IEEE International Conference on Coputer Vision and Pattern Recognition, 8, pp. -8. [] J. Liu, J. Luo, and M. Shah, Recognizing realistic actions fro videos in the wild, in Proc. the IEEE Conference on Coputer Vision and Pattern Recognition, 9, pp [] A. Gilbert, J. Illingworth, and R. Bowden, Fast realistic ultiaction recognition using ined dense spatio-teporal feature, in Proc. the International Conference on Coputer Vision, 9, pp [3] W. Brendel and S. Todorovic, Activities as tie series of huan postures, in Proc. European Conference on Coputer Vision,, pp [4] A. Kovashka and K. Grauan, Learning a hierarchy of discriinative space-tie neighborhood features for huan action recognition, in Proc. the IEEE Conference on Coputer Vision and Pattern Recognition,, pp [5] Q. Le, W. Zou, S. Yeung, and A. Ng, Learning hierarchical invariant spatio-teporal features for action recognition with independent subspace analysis, in Proc. the IEEE Conference on Coputer Vision and Pattern Recognition,, pp [6] B. Li, M. Ayazoglu, T. Mao, and O. Caps, Activity recognition using dynaic subspace angles, in Proc. the IEEE Conference on Coputer Vision and Pattern Recognition,, pp Jiangfeng Yang was born in Yunnan Province, China, in 975. He received the B.S. degree in applied physics fro the University of Yunnan, Kuning, in 997, and the Master of engineering degree in coputer software and theory fro the Kuning University of Science and Technology, Kuning, in 9. He is currently pursuing the Ph.D. degree with the Departent of Counication and Inforation Engineering, University of Electronic Science and Technology of China (UESTC). His research interests include action recognition, object recognition and classification. Zheng Ma was born in Chengdu Province, China, in 956. He received the B.S. degree fro UESTC, in 98. He is a professor and Vice President of UESTC. His research interests include signal processing, inforation security, iage processing, spectral estiation, array signal processing, and inforation theory. 4 Engineering and Technology Publishing 9