Directionally-grouped CHLAC Motion Feature Extraction and Its Application to Sport Motion Analysis

Transcription

1 Directionally-grouped CHLAC Motion Feature Extraction and Its Application to Sport Motion Analysis FUMITO YOSHIKAWA, TAKUMI KOBAYASHI, KENJI WATANABE, NOBUYUKI OTSU National Institute of Advanced Industrial Science and Technology Umezono, Tsukuba, JAPAN Abstract: - In this paper, we propose a method for extracting motion direction of objects by directionally-grouped cubic higher-order local auto-correlation (DG-CHLAC) motion features. The DG-CHLAC motion features are obtained by classifying CHLAC feature components into predefined direction groups. By identifying the dominant feature component in DG-CHLAC with respect to magnitude, direction of motions can be simply detected. The experimental results on different motions in sport confirmed the effectiveness of the method. Key-Words: video based motion analysis, detection, segmentation, feature extraction, CHLAC, motion direction 1 Introduction Appearance based sport motion analysis has attracted a great deal of attentions, particularly in high performance sport [1, 2]. The sport motion analysis is mostly conducted on repetitively performed sport motions, with requiring large amount of videos to be processed. Detecting and segmenting sport motions of interest in videos is a fundamental procedure for understanding the motions and then utilizing the video contents, e.g. to facilitate sport coaching. It is also necessitated in various applications such as quantitative motion analysis, performance evaluation, and video browsing. However, ongoing routines with kinematical analysis of sport motions involve heavy manual labor in both the data processing and handling [1-4]. For example, operators have to manually seek and isolate target activities in videos and digitize object points of interest, e.g. human joint centers. In addition, existing approaches employed in kinematical analysis of sport motions have more or less difficulties in feeding back the analysis results for coaches and athletes timely according to their requirements. Therefore, it is expected that computer vision and pattern recognition techniques contribute to automating the video analysis. In recent years, various motion feature extraction methods have been developed for the tasks to detect and recognize the full-body human motions and also applied to sport-related tasks [5-9]. It is obviously desirable that a method to be developed has general versatility. However, many of ordinary approaches require specific and tedious steps including segmentation of the player regions, in which the error tends to affect final recognition. On the other hand, a scheme of adaptive vision system employing cubic higher-order local auto-correlation (CHLAC) has been presented by Kobayashi and Otsu [10, 11], and some of the authors have successfully applied to the wide range of video analysis; for example, motion recognition [11], detection [12-14], segmentation [15] and evaluation [16]. CHLAC features characterize object motion, reflecting kinematical measures such as velocity and acceleration as well as variability of motion patterns in the various directions. In addition, CHLAC requires neither prior knowledge, heuristics about objects nor expensive computational cost. Such the properties of CHLAC are preferable for motion analysis even in each sport, and CHLAC can be widely applied to sport-related tasks. In this paper, we propose a directionally-grouped CHLAC motion feature extraction method which characterizes object motions in terms of direction. The proposed method can automatically detect the dominant direction of motions in robust manner. The experimental results on different disciplines of sport motions confirmed the effectiveness of the method. 2 Directionally-grouped CHLAC In this section, we describe the details of the proposed directionally-grouped CHLAC. First, a method of extracting CHLAC motion features and its properties are described briefly (For details see [11]). Next, the ISBN:

2 r + a 1 1 λ r + a 2 Fig. 1. Examples of images in the preprocessing of feature extraction; an original horizontal bar exercise image captured by a stationary camera and its binary image. way of grouping CHLAC components in terms of directions is presented and then the method to detect the motion direction is also described. 2.1 Preprocessing In the preprocessing stage, the input color image sequences are converted to binary valued images by applying Otsu s automatic threshold selection method [17] to the histogram of inter-frame difference values. These processes filter out both noise and illumination changes, which are irrelevant to both posture and the motion of objects, i.e. player and their sport gears. The resultant pixel values in each frame are either 1 (moved) or 0 (static). Fig. 1 shows an example of input videos captured by a stationary camera, and the binarized images corresponding to each input. 2.2 CHLAC CHLAC extracts motion features as histograms of spatiotemporal local configuration patterns of moving points (pixels) found by the aforementioned preprocessing. Let f (r) be three-way data (cubic data) with r= (x, y, t) T defined on the image sequences V : X Y T, where X and Y are the width and height of image frame and T is the length of a time-window. The N-th order auto-correlation can be defined as, x N ( t; a 1, a 2 L, a N ) = f ( r ) f ( r + a ) f ( r + a ) f ( r + a 1 2 L N ) d r V, (1) where a i (i = 1,,N) are displacement vectors from a reference point r. CHLAC features characterize object motions in terms of gradients and curvatures in a frame image as well as velocities and accelerations along time axis. CHLAC features also reflect various motion patterns in the diverse directions. CHLAC enables simultaneous extraction of spatiotemporal features from the motion images without prior knowledge nor λ Fig. 2. Examples of mask patterns: (left) N=0; (middle), N = 1, a 1 = ( λ, λ,1) T ; (right), N = 2, a 1 = ( λ, λ, 1) T, a 2 = ( λ, λ,1) T heuristics about objects and also without segmenting and tracking each target object by virtue of shift-invariance and additivity. Moreover, its computational cost is very low due to simple sums of products. The proposed method inherits these favorable properties which benefit detection of motion direction as well. 2.3 Directionally-grouped CHLAC Although CHLAC can extract various patterns of motions by many mask patterns, it is difficult to interpret the extracted features. We further extract meta-features over CHLAC from the viewpoint of motion directions so as to qualitatively comprehend the feature. Directionally-grouped CHLAC motion features (hereafter simply called, DG-CHLAC features) are obtained by classifying and integrating CHLAC components into predefined directions (say 8-directions). The classified mask patterns are shown in Fig 3. The procedure to extract DG-CHLAC is straightforward as follows. First, 8 feature components within each direction group (in Fig. 3) are summed up; x ( t ) = x t; a, a,, a ( ) Di 1 2 L ( a, a L, a ) Di i 2 N N, (2) where x Di is a DG-CHLAC feature of the direction Di ( i = 1,2, L,8 ). The dimensionality of DG-CHLAC features amounts to 8. So computed DG-CHLAC features are local directional features at each pixel, and those are summed up over a certain region (say, whole image) to provide global directional features. 2.4 Detection of motion direction In the DG-CHLAC features, the higher the value of x Di, it is more likely that target objects are moving relatively in the direction of Di. Thus, the dominant motion direction is detected as follows, ISBN:

3 γ k i = x j P Di x Dj = max, (3) arg γ i, (4) i P where P indicates the direction subset of interest, such as P={D1,D5} (vertical direction). The index γ i is regarded as the probability across the direction subsets. 3 Experimental Results In the experiment, we applied the DG-CHLAC method to the two tasks related to sport motion analysis; motion segmentation and detection. D1 D2 D3 D4 3.1 Motion segmentation For motion segmentation, we used horizontal bar exercise videos captured by a stationary camera. The dataset contains time-varying image sequences of 7 single series of horizontal bar exercises performed by 6 well-trained athletes. In each video, sport actions such as swinging, and release and re-grasps, and dismounting are recorded. In addition background noises derived from other moving persons are found. The total number of frames is The size of image frames is , and the frame rate is about 30 fps. In this type of sport motions, the dominant motion directions are upward and downward, and thus we only focus those directions, i.e., P= {D1, D5} in Eq. (3), (4). Those two directional features can discriminate the motions. Fig. 4 shows the result of upand down-motion phases segmented by the proposed method. The values of line chart located below the image represent the value of γ D1 (upward direction-grouped CHLAC components). The regions t-1 t t+1 t-1 t t+1 t-1 t t+1 t-1 t t+1 D5 D6 D7 D8 t-1 t t+1 t-1 t t+1 t-1 t t+1 t-1 t t+1 Fig. 3. Directionally-grouped mask patterns Fig. 4. Experimental results of motion segmentation. Up- (magenta) and down- (cyan) motion phases. Each frame image at the time indicated by the red vertical bar is displayed. The magenta and cyan bars at the bottom indicate the ground truth periods. ISBN:

4 colored in magenta correspond to up-motion phase (γ D1 > 0.5) and those colored in cyan correspond to down-motion phase (γ D1 < 0.5). As shown in Fig. 4, the proposed method yields favorable results. 3.2 Motion detection For motion detection, we used vault exercise videos captured by a panning camera. The dataset contains time-varying image sequences of 6 single series of vault exercises performed by 6 well-trained athletes. In each vault, the gymnast begins with a run and a take off the vault board by the both feet, followed by a brief support phase on the table with two hands. In this experiment, the camera recording those videos is first panned and then remains stationary. Fig. 5 shows example of an original image captured while the camera is panned and its binary image. It can be said that it is a challenging task to detect target motions in noisy image sequences captured by a moving camera. The total number of frames is The size of image frames is , and the frame rate is about 30 fps. By considering the panning movement of the camera, we also focus on the horizontal direction (rightward and leftward) by P= {D3, D7}, in addition to the vertical direction as in Sec.3.1. The values of bar chart located below the image represent γ D1 (upward) and γ D3 (rightward). The regions colored in magenta correspond to up-motion phase (γ D1 > 0.5) and those colored in cyan corresponds to down-motion phase (γ D1 < 0.5). As shown in Fig. 6 (b), the proposed method based on γ D1 can detect the motion that the gymnast is hopping up, from the panning movement that dominate the motion flows. This result shows the robustness of the proposed method to measure the motion directions by the meta-features (DG-CHLAC). As shown in Fig.6 (a) and (c), the panning movement, especially start (a) and end (c), can also be recognized by the outstanding drop and overshoot of values γ D3. (a) (b) Fig. 5. Example of an original vault exercise image (left) captured by a panning camera and its binary image (right). (c) Fig. 6. Experimental results of motion detection. The red vertical line and the corresponding image is at the fame (a) where the camera starts to be panned, (b) where the gymnast is hopping up in front of the vault board and (c) where the camera panning comes to the end. The black bars indicate no dominant motion phase in vertical direction. ISBN:

5 4 Conclusion We have proposed a directionally-grouped CHLAC motion feature extraction method which characterizes object motions in terms of direction. The experimental results exhibit potential ability of the proposed method to detect the motion direction. By incorporating motion recognition techniques, the proposed method will be utilized for more refined and higher-level video analysis, such as characterization of each motion and also skill. References: [1] Gideon Ariel, Sports Technology from Mexico City Olympic 1968 to the Future Olympics in Beijing 2008, Proceedings of Computer Science in Sports, pp. 2-14, 2007 [2] Hughes, M. Hughes, M.T., and Behan, H., The Evolution of Computerised Notational Analysis through the Example of Racket Sports, In International Journal of Sports Science and Engineering, Vol. 1, No. 1, pp.3-28, 2007 [3] Yeadon, M. R., and Hiley, M. J., The mechanics of the backward giant circle on the high bar, Human movement science, 19, pp , 2000 [4] Irwin, G., Kerwin, D., and Samuels, M., Biomechanics of the longswing preceding the Tkachev, Proc. XXV ISBS symposium pp , 2007 [5] Bobick, A. F., Davis, J. W., The Recognition of Human Movement Using Temporal Templates, In: IEEE Transactions Pattern Analysis and Machine Intelligence, Vol. 23, No.3, pp , 2001 [6] Zivkovic, Z., van der Heijden, F., Petkovic, M., Jonker, W., Image Segmentation and Feature Extraction for Recognizing Strokes in Tennis Game Videos, In: Proceedings 7th Annual Conference on the Advanced School for Computing and Imaging, 2001 [7] Zhu, G., Xu, C, Huang, Q, Gao, W, Xing L., Player Action Recognition in Broadcast Tennis Video with Applications to Semantic Analysis of Sports Game, In: the 14th annual ACM international conference on Multimedia, 2006 [8] Zelnik-Manor, L., Irani, M., Statistical Analysis of Dynamic Actions. In: IEEE Transactions Pattern Analysis and Machine Intelligence, Vol. 28, No. 9, pp , 2006 [9] Roh, M.C., Christmas, B., Kittler, J., Lee, S.W., Gesture Spotting for Low-resolution Sports Video Annotation, Pattern Recognition 41, pp , 2008 [10] Otsu, N., 2008, CHLAC Approach to Flexible and Intelligent Vision Systems, In: ECSIS Symposium on Bio-inspired, Learning, and Intelligent Systems for Security. [11] Kobayashi, T., Otsu, N., Three-way Auto Correlation Approach to Motion. Recognition, Pattern Recognition Letter, Vol. 30, No. 3, 2009, pp [12] Nanri, T., Otsu N.: Unsupervised Abnormality Detection in Video Surveillance. In: IAPR [13] Iwata, K., Satoh, Y., Sakaue K., Kobayashi, T., Otsu, N.: Development of Software for Real-time Unusual Motions Detection by Using CHLAC. In: ECSIS Symposium on Bioinspired, Learning, and Intelligent Systems for Security, (2008) [14] Yoshikawa, F., Kobayashi, T., Watanabe, K., Shirai, K., Otsu, N., 2010, Start and End Point Detection of Weightlifting Motion Using CHLAC and MRA, In: 1st International Workshop on Bio-inspired Human-Machine Interfaces and Healthcare Applications, 2010, pp [15] Kobayashi, T., Yoshikawa, F., Otsu, N., Motion Image Segmentation Using Global Criteria and DP, In: International Conference on Automatic Face and Gesture Recognition, [16] Yoshikawa, F., Kobayashi, T., Watanabe, K., Otsu, N., CHLAC based Index for Evaluating Inefficiency of Sport Motions, Proc. of 3rd Annual International Conference Physical Education Sport and Health, pp , 2010 [17] Otsu, N., A threshold selection method from gray-level histogram, In: IEEE Transactions on System Man Cybernetics, Vol. SMC-9, No. 1, 1979, pp ISBN: