Kinect Joints Correction Using Optical Flow for Weightlifting Videos

Transcription

1 215 Seventh International Conference on Computational Intelligence, Modelling and Simulation Kinect Joints Correction Using Optical Flow for Weightlifting Videos Pichamon Srisen Computer Engineering Faculty of Engineering, Sansanee Auephanwiriyakul Senior Member, IEEE Computer Engineering Faculty of Engineering, Nipon Theera-Umpon Senior Member, IEEE Electrical Engineering Faculty of Engineering, Samatchai Chamnongkich Physical Therapy Faculty of Associated Medical Science, Abstract To ease a coach in weightlifting training, automatic weightlifting pattern evaluation is required. In order to do that, the motion tracking process is always needed. Kinect sensor is one of the popular sensors for that. However, there is a problem with skeleton created by the Kinect sensor because of self-occlusion. Hence, in this paper, we develop a joint correction process for 3 types of joints including hands, feet, and knees since these joints are sometimes provided incorrectly. However, we only correct these joints in the the first pull to the transition from the first to the second pull (first-step) in snatch, and clean and jerk weightlifting and the turnover under the barbell to the catch phase (second-step) in clean and jerk weightlifting. This is because miscalculation occurs only in these steps. We utilized fast cross-correlation and the Lucas-Kanade algorithm to compute the optical flow of the consecutive frames. From that, we then correct the joints if they are misplaced from the predicted joints. Our system provides better joints and more preferable to human eyes than the original Kinect skeleton. better to avoid using 2 Kinect sensors or attaching a sensor with a weightlifter. Keywords Snatch weightlifting; Clean and jerk weightliftin; Optical flow; Lucas-Kanade algorithm; Kinect joints; Fast crosscorrelation I. INTRODUCTION Nowadays, there are many applications using human motion detection. One application of human motion detection is in weightlifting training [1 3]. Since human motion detection is able to help improve the performance of a weightlifter and also help a coach in pinpointing a wrong movement easily. A widely used sensor in the human motion detection algorithms is the Kinect sensor [4, 5]. Not only the sensor can provide color images, but it also creates range images using an infrared camera [6]. A human skeleton with 2 joints is also created from the sensor as well. Each joint is a position in 3-dimensional space (x,y,z) where z is the depth information. However, the generated skeleton might be incorrect because of self-occlusion. Therefore, a joint correction algorithm is needed before a weightlifting pattern evaluation can be calculated. There are a few algorithms trying to correct the joints [7, 8]. However, both algorithms utilize a fusion method that fuses the data from either 2 Kinect sensors or a Kinect sensor and a wearable inertia sensor. Although, these algorithms provide a good result, it is Figure 1. Rframes of first-step in Snatch and Clean and jerk, and Second-step in clean and jerk Therefore, in this paper, we propose an algorithm to correct joints for weightlifting program using only one Kinect /15 $ IEEE DOI 1.119/CIMSim

2 sensor. In particular, we utilize Lucas-Kanade template tracking [9] to help in optical flow computation. Each predicted joint from the optical flow is compared with that provided by Kinect. If the provided joint is far from the supposed position, we then correct that joint. We compare the resulting joint positions with an expert s opinion. We also ask the preference between corrected joints and provided joints from 3 volunteers. We implement our algorithm on the data set collected from the first pull to the transition from the first to the second pull (first-step) in snatch, and clean and jerk weightlifting and the turnover under the barbell to the catch phase (second-step) in clean and jerk weightlifting. This is because these two steps create incorrect joints more often. Moreover it is shown that these steps are the important steps in snatch, and clean and jerk weightlifting [1, 11]. Representative frames (Rframes) of the first-step of snatch, and clean and jerk weightlifting styles and Rframes of the second-step of clean and jerk weightlifting are shown in figure 1. II. SYSTEM OVERVIEW A Kinect sensor is set 3 meters in front of a weightlifter with.8 meters above the ground and angle of elevation. The Kinect sensor provides 3 sets of data, i.e., skeleton information, RGB image, and shape image (black-white (BW) images) at a time. To speed up our process, we downsample each image to by using bicubic interpolation [12]. In each skeleton image, there are 2 joints provided by the Kinect sensor, i.e., left ankle, right ankle, left elbow, right elbow, left foot, right foot, left hand, right hand, head, left hip, right hip, center hip, left knee, right knee, left shoulder, right shoulder, center shoulder, spine, left wrist, and right wrist. Although, each joint is provided in the form of (x,y,z) coordinate, we only use the coordinate in x-y plane (x,y). We use only 16 joints in our calculation since we do the correction on the joints that are superimposed on the shape image. The left and right hands will be collapsed into left and right wrists. The left and right feet are collapsed into left and right ankles. Please be noted that we call left and right parts of the skeleton according to the left and right parts of the actual body. Figure 2 shows the joints used in the experiment. In our experiment, we superimpose joint coordinates into the shape image and then we use image coordinate in the remaining steps to ease our calculation. From the way we set up the Kinect sensor, the spine joint coordinate is approximately (,). Therefore, we first find the center of the body and use that as a spine joint. Then we build the 15 remaining joints with respect to the spine joint based on the skeleton information. We convert each RGB image into a luminance image using [13] I( xy, ) =.2989 R( xy, ) G( xy, ) B( xy, ) (1) Then the background area of an RGB image is removed by superimposing the black region of a BW image onto the corresponding area in gray scale image as shown in figure 3. The fast normalized cross correlation [14] is then used to find hands, feet, knees, and weight plates in the starting frames of each weightlift style. Please be noted that we only correct the joints in the sequences of the first-step in snatch, and clean and jerk weightlifting styles and the second-step in clean and jerk weightlifting style. Because only some incorrect joints occur in these steps, template matching will be performed on the first frame of each sequence. Head Right Shoulder Center Shoulder Right Elbow Left Shoulder Spine Right Hip Left Elbow Right Wrist Left Wrist Left Hip Center Hip Right Hand Left hand Right Knee Left Knee Right Foot Left Ankle Right Ankle Left Foot Figure 2. Sixteen Kinect joints used in the experiment shown in black and 4 unused Kinect joints in the experiment shown in red Now we will briefly describe the template matching using the fast cross-correlation. Let T be a template with the size of K L, the correlation coefficients of the template with subimage in the first image frame I with the size of M N of each sequence is computed by [14] I( xy, ) Ii, j T( x iy, j) T xy, R( i, j) = I( xy, ) I i, j T( x iy, j) T xy, xy, (2) where i {,1, M 1}, j {,1, N 1}, T is the mean of the template T and I i, j is the mean of the subimage coinciding with the template T. Since R(i,j) is normalized, its values will be [ 1,1]. The maximum value will point to the most similar subimage to the particular template. Figure 3. Removing image background: BW image Gray scale image Image I(x,y) after background removal 27

3 (e) Figure 4. Template images T: Hands in snatch, Hands in clean and jerk Feet (d) Knees, and (e) Weight plates (d) The templates used in this step are 1 templates in total for both weightlifting styles. There are 2 hand-templates (both with the size of 2 2) for snatch weightlifting. For clean and jerk weightlifting, the utilized templates are 2 handtemplates (both with the size of 23 23), and 2 weight plate templates. Also, for both weightlifting styles, we use 2 feettemplates (both with the size of 23 25) and 2 kneetemplates (both with image size). These templates are shown in figure 4. We manually selected each template from 1 athlete in both snatch and clean and jerk weightlifting videos. Figure 5 shows the example of subimage matched with each template in both weightlifting styles. For the consecutive frames in the image sequence, we find hands, feet, knees and weight plates using the Lucas- Kanade optical flow [9]. The optical flow is computed between the image at time t and time t+1 by aligning extracted subimage T(x,y) from cross-correlation (previous step) to image I(x,y). Example of T in I at time t and t+1 is shown in figure 6. Now, we briefly describe the Lucas- x+ p1 Kanade algorithm. Let W( x; p ) = y+ p be the 2 parameterized set of allowed warps where p =(p 1, p 2 ) is the optical flow. The aim is to minimize the objective function of the Lucas-Kanade algorithm as follows: min I( W( x; p) ) T( x ) (3) p x where x =(x,y). Since the pixel values in I are unrelated to the coordinate, the optimization in equation (3) will be changed to min I( W( x; p+δp) ) T( x ) p (4) x by assuming that a current p is known. p is iteratively solved by updating the following equation p = p+δp (5) These two steps (equations 4 and 5) are iterated until the estimated p is converged. x I(x) x1,y1 x1, y1 y x2', y2' T(x) x2, y2 Figure 6. Example of T in image I at time t with coordinate (x1,y1) and at time t+1 with coordinate (x1,y1 ) Figure 5. Example of matching in hands, feet, knees, and weight plates of Snatch and Clean and jerk weightlifting styles (Note we do not find weight plates in snatch weightlifting style) (d) Figure 7. Pixels in the templates that are selected (indicated by asterisks) to represent hands, feet, knees, and (d) weight plates. 28

4 Figure 8. I at time t, I at time t+1, and Optical flow shown in pink arrows. Skeleton (x,y) joints Euclidean Adjust joints if the distance is bigger than the threshold Kinect RGB image, Shape image Convert RGB image to gray Background Template matching in the first frame of each Finding optical flow using Lucas- Kanade algorithm (x,y) tracking Corrected joints Figure 9. Joint correction process Although, the Lucas-Kanade algorithm provides optical flow for the whole subimage, we are interested in only the points corresponding to the joint that we want to correct. Hence to represent the hands and feet in the first-step of both weightlifting styles, we use the point at the middle of the template. However, to represent the knees, we use the point at the top middle of the template. In the first-step and second-step of clean and jerk weightlifting, we use the middle pixel of the template to represent the weight plates. Figure 7 shows the points we pick in hands, feet, knees, and weight plates. Figure 8 shows an example of optical flow of the points of interest. However, for the second-step in clean and jerk weightlifting, the hand part is difficult to find. Therefore, we calculate the distance between each hand and its corresponding weight plate from the first frame in the firststep. When we need to find the hand part in the frame in the second-step, we calculate back where the hand is supposed to be from the kept distance and the pixel representing the weight plate in that particular frame. After, we get all of the representing pixels, we calculate the Euclidean distance between Kinect hand joints, feet joints, and knee joints with the pixel representing hands, feet, and knees in the first-step of both weightlifting styles. Please be noted that since we superimpose the Kinect joints into gray scale image, now the coordinate of joints will be the pixel coordinate. If the Euclidean distance is bigger than a threshold, the Kinect joint coordinate will be changed to the pixel coordinate representing that part. In the secondstep of clean and jerk weightlifting, only hand joints are incorrect, we hence correct only these parts using the previously mentioned scheme. In the experiment, the threshold is manually selected to 2.5. The joint correction process is shown in the diagram in figure 9. III. EXPERIMENTAL RESULTS We collected data set from 3 athletes (1 female and 2 males) who perform 1 snatch and 1 clean and jerk. Hence there are 3 videos for each weightlifting style. We then selected consecutive frames for the first-step from snatch, and clean and jerk and the second-step from clean and jerk weightlifting. Tables 1 and 2 show the number of frames from all 3 athletes used in the experiment. There are 2969 and 2563 frames in the first-step of snatch, and clean and jerk, respectively. There are 4963 frames in the second-step of clean and jerk. TABLE I. NUMBER OF FRAMES USED IN THE FIRST-STEP OF SNATCH. Video First- First- Total Total Total First To validate our result, we ask an expert to correct the joints and we call that a ground truth. Then the mean square error (MSE) between automatically corrected joints and the corresponding joints from the ground truth is calculated as follows: 1 N i i i i 2 2 ( ) ( ) (6) MSE = s x + t y N = j 1 i joints 29

5 where (s i,t i ) and (x i,y i ) are the coordinates of the i th joint from the ground truth and from the correction process, respectively. Vide o TABLE II. Tot al NUMBER OF FRAMES USED IN THE FIRST-STEP AND SECOND-STEP OF CLEAN AND JERK. First Secon First Secon First Secon Tot Tot - d- - d- - d- al al TABLE III. MSE (IN PIXELS) OF THE FIRST-STEP IN SNATCH WEIGHTLIFTING OF JOINTS BEFORE AND AFTER CORRECTION. Video Before After Before After Before After average TABLE IV. MSE (IN PIXELS) OF THE FIRST-STEP AND SECOND- STEP IN CLEAN AND JERK WEIGHTLIFTING OF JOINTS BEFORE AND AFTER CORRECTION. Video Before After Before After Before After average TABLE V. AVERAGE OF MSE OF JOINTS BEFORE AND AFTER CORRECTION IN BOTH SNATCH, AND CLEAN AND JERK WEIGHTLIFTING STYLES. Snatch Clean and jerk Before After Before After We also ask 5 subjects to look at the corrected joints that are superimposed into the gray scale images. They have to grade the quality of the corrected joints in the scale 1 to 1, where 1 means the best and 1 means the worst. Figure 1 shows the examples of the automatically corrected joints. Tables 3, 4, and 5 show the MSE results for both snatch, and clean and jerk, whereas tables 6, 7, and 8 show the scores collected from 5 subjects. Skeleton Information Incorrect Joints Corrected Joints Figure 1. Incorrect Kinect joints and corrected joints from the algorithm We can see that from tables 3, 4, and 5, the MSE of the corrected joints are lower than those of the Kinect joints significantly. The average MSE of the corrected joints in snatch is 4.99 pixels, and that in clean and jerk is 5.93 pixels. Whereas the average MSE of the original Kinect joints in snatch, and clean and jerk are and pixels, respectively. The preference from 5 subjects is also the corrected joints as we can see from tables 6, 7, and 8. The average scores of the corrected joints in snatch, and clean and jerk are 8.14 and 7.97, respectively. Whereas that of the original Kinect joints in snatch, and clean and jerk are 2.73 and 4.65, respectively. Although, the correction process provides good results and they are preferable by humans, there are still some mistakes occurring from the Lucas-Kanade optical flow algorithm. For example, in snatch weightlifting, there might be the hand tracking which is a little bit far off the actual 3

6 hand, hence, the hand coordinate is calculated wrongly as shown in figure 11. Another example is an error from clean and jerk weightlifting as shown in figure 11, when the optical flow detects the weight plate slightly off its actual center. Hence the weight plate coordinates will be wrongly calculated, then the hand coordinates will be wrong also. TABLE VI. SCORE GIVEN BY EACH SUBJECT ON SNATCH SKELETONS BEFORE AND AFTER CORRECTING JOINTS. Subjects Before After Before After Before After average TABLE VII. SCORE GIVEN BY EACH SUBJECT ON CLEAN AND JERK SKELETONS BEFORE AND AFTER CORRECTING JOINTS. Subjects Before After Before After Before After average TABLE VIII. AVERAGE SCORE FOR SNATCH, AND CLEAN AND JERK SKELETONS BEFORE AND AFTER CORRECTING JOINTS. Snatch Clean and jerk Before After Before After Figure 11. Examples of wrongly coordinate calculation in Snatch and Clean and jerk IV. CONCLUSION Recently, there is a need for automatic weightlifting training, to help a coach to improve his/her weightlifter. One of the methods to do that is to use motion detection. One popular sensor used in this area is Kinect sensor. However, there are some problems with the skeleton joints provided by Kinect sensor from self-occlusion. Therefore, in this paper, we develop a process to correct 3 joints, i.e., hands, feet, and knees in both snatch, and clean and jerk weightlifting styles, since these parts are mostly incorrect. There is one step in snatch that is needed to be corrected, i.e., the first pull to the transition from the first to the second pull (first-step). There are two steps in clean and jerk weightlifting that are required the correction, i.e., the first-step and the turnover under the barbell to the catch phase (second-step). We utilize the fast cross-correlation and the Lucas-Kanade algorithm to find the optical flow between the consecutive frames and correct the joints when the original joint is far off the calculated ones. The results show that our correction process provides a better result and more preferable to human than the original Kinect joints. ACKNOWLEDGMENT The authors would like to thank athlete volunteers at the weight training SugSaSongKrua Chiang Mai School, Mae Rim District, for the weightlifting dataset information. REFERENCES [1] J. Lei, M. Liu, J. Ma, Q. Song, L. Qiu and Y. Ge, Movement pattern recognition of weight lifter based on ground reaction force, Proceedings of the 6th World Congress on Intelligent Control and Automation, Dalian, China, 26, pp [2] H. Novatchkov and A. Baca, Artificial intelligence in sports on the example of weight training, Journal of Sports Science and Medicine, 213, vol.12, pp [3] J. Walsh, J. Quinlan, R. Stapleton, D. FitzPatrick and D. McCormack, Three-dimensional motion analysis of the lumbar spine During Free Squat weightlift training, The American Journal of Sports Medicine, 27, Vol. 35, pp [4] M. Ye, X. Wang, R. Yang, L. Ren and M. Pollefeys, Accurate 3d pose estimation from a single depth image, Proceedings of the 211 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 211, pp [5] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman and A. Blake, Real-Time Human Pose Recognition in Parts from Single Depth Images, Machine Learning for Computer Vision Studies in Computational Intelligence, 213 vol. 411, pp [6] J. Webb and J. Ashley, Beginning Kinect Programming with the Microsoft Kinect SDK, in Apress, New York, 212 [7] K. Yeung, T. Kwok and C. C. L. Wang, Improved Skeleton Tracking by Duplex Kinects: A Practical Approach for Real-Time Applications, Journal of Computing and Information Science in Engineering, 213, vol. 13, pp. 1. [8] F. Destelle, A. Ahmadi, N. E. O'Connor, K. Moran, A. Chatzitofis, D. Zarpalas and P. Daras, Low-Cost Accurate Skeleton Tracking Based on Fusion of Kinect and Wearable Inertial Sensors, Proceedings of the 22nd European Signal Processing Conference (EUSIPCO 214), Lisbon, Portugal, 214, pp [9] S. Baker and I. Matthews, Lucas-Kanade 2 Years On: A Unifying Framework, International Journal of Computer Vision, 24, vol. 56, pp [1] E. Harbili and A. Alptekin, Comparative kinematic analysis of the snatch lifts in elite male adolescent weightlifters, The Journal of Sports Science and Medicine, 214, pp [11] M. Rippetoe, The Power Clean, The CrossFit Journal Article Reprint, 26, pp [12] R. C. Gonzalez and R. E. Woods, Digital Image Processing (Third Edition), in Pearson Education Inc, 28. [13] R. C. Gonzalez and R. E. Woods, Digital Image Processing (Third Edition), in Pearson Education Inc,1992. [14] J. P. Lewis, Fast Normalized Cross Correlation, Vision Interface, vol. 1, issue. 1,