Reproduction of Human Arm Movements Using Kinect-Based Motion Capture Data

Transcription

1 Reproduction of Human Arm Movements Using Kinect-Based Motion Capture Data José Rosado, Filipe Silva, Vítor Santos and Zhenli Lu, Member, IEEE Abstract Exploring the full potential of humanoid robots requires their ability to learn, generalize and reproduce complex tasks that will be faced in dynamic environments. In recent years, significant attention has been devoted to recovering kinematic information from the human motion using a motion capture system. This paper demonstrates and evaluates the use of a Kinect-based capture system that estimates the 3D human poses and converts them into gestures imitation in a robot. The main objectives are twofold: (1) to improve the initially estimated poses through a correction method based on constraint optimization, and (2) to present a method for computing the joint angles for the upper limbs corresponding to motion data from a human demonstrator. The feasibility of the approach is demonstrated by experimental results showing the upper-limb imitation of human actions by a robot model. P I. INTRODUCTION ROGRAMMING robots to perform complex tasks and extends its repertoire can be extremely tedious and time consuming. Learning from demonstration is a promising methodology that offers a more intuitive approach to teach a robot how to generate its own motor skills [1, 2]. To this end, the robot should be able to estimate human poses when performing a desired task, as well as to translate the skeleton data into appropriate motor commands. In the last years, a large body of work has studied the use of marker based motion capture systems for extracting 3D poses as input for training robots to perform complex motions [3-6]. Despite much research progress, these systems are usually expensive, they require careful calibration and its application is limited to rigid environments. To overcome these limitations, the main challenge is to develop accurate methods for extracting 3D human poses from image sequences using low-cost systems. as a valid alternative. Recently, the field of markerless motion capture has experienced a strong evolution with the development of high-speed and cheap depth cameras. In particular, the depth data provided by the PrimeSense sensor opened up new opportunities for extracting gesture-based interactions with a more portable and less costly system. The publica- J. Rosado is with the Department of Computer Science and Systems Engineering, Coimbra Institute of Engineering, IPC, Coimbra, Portugal (jfr@isec.pt). F. Silva is with the Department of Electronics, Telecommunications and Informatics, Institute of Electronics and Telematics Engineering of Aveiro, University of Aveiro, Portugal ( fmsilva@ua.pt). V. Santos is with the Department of Mechanical Engineering, Institute of Electronics and Telematics Engineering of Aveiro, University of Aveiro, Portugal ( vitor@ua.pt). Z. Lu is with the Institute of Electronics and Telematics Engineering of Aveiro, University of Aveiro, Portugal ( zhenli.lu@ua.pt). tion of the tracking algorithm of the Kinect Software Development Kit [4] and the availability of several development environments (e.g., Microsoft SDK) have contributed for a growing interest in model-free approaches. However, the success of these alternatives depends on the accuracy and robustness required in each specific area of application. This paper addresses the main concern associated with the use of a Kinect-based human motion capture in robotics: the lack of a kinematic model to assure coherence in the provided poses. The main objective is to demonstrate and evaluate both a human action pose correction method and an inverse kinematics technique. The former aims to assure constant limb lengths over an entire sequence of poses. The later converts each of the 3D poses into the corresponding angles for the upper-body joints, including a validation test to deal with physical limits (e.g., joint limits). The motivation of this work is to create a database of classified motions to learning control in robotics. In line with this, the remainder of the paper is organized as follows: Section II presents the motion capture system based on a single Kinect camera and the experimental conditions. Section III describes the pose correction method based on constraint optimization. Section IV focuses on the kinematic mapping from 3D poses to joint angles. Section V discusses the results achieved to validate the proposed solutions. Finally, Section VI concludes the paper and proposes future extensions. II. HUMAN MOTION CAPTURE The Kinect sensor provides a depth image, at 30 frames per second, for the skeleton-based pose estimation with depth resolution of a few centimeters. The human skeleton estimated from the depth image includes a total of 20 body joints that will be the input for our approach. These captured data consists of a set of Cartesian points in the 3D volume for each human pose, which will be called raw-data hereinafter. Several studies have assessed the accuracy of the depth reconstruction and joint positions from the Kinect pose estimation, including comparisons with ground truth motion capture data [9-11]. In general, these studies highlight the potential of the Kinect skeleton in controlled body postures whenever self-occlusions are avoided. In the experiments, we have used a single Kinect camera positioned at about 3 meters from the human subject to capture the whole body standing upright. The human pose estimation is fully automatic and did not require calibration.

2 Fig. 1. Frame-to-frame variation of the limb lengths for a static posture (left) and a reaching arm movement (right). In this study the attention is dedicated to the upper limbs, including the shoulder, elbow and wrist joints of both right and left arms. In order to ensure the most convenient acquisition conditions, the human subject was asked to prevent lower trunk movements and to perform controlled scapular motions. Precautions were also taken to avoid occlusions of the upper limb parts. Besides the accuracy and robustness of the skeletal poses, a critical element is the stability of the estimated frameto-frame body geometry. As mentioned before, a characteristic of the human body skeletonization with the Kinect sensor is that the limb lengths are not kept constant through the entire sequence and differ between the two arms. Fig. 1 illustrates the variations of the limb lengths, from frame-toframe, a static posture and a reaching arm movement were evaluated. In the static case, the mean value rounds 268 mm for the arm and 233 mm for the forearm, while the standard deviation is around 3.65 mm and 1.51 mm, respectively. These measures are significantly different during the execution of a reaching movement: 265 mm of mean for the arm, 216 mm for the forearm with a standard deviation around 15.9 mm and 8.8 mm, respectively. III. CONSTRAINED-BASED MOTION FILTERING The pose correction method aims to convert the motion of a source human subject into a new motion, while satisfying a given set of kinematic constraints. These kinematic constraints are formulated in order to assure a kinematic model with constant limb lengths. The proposed method, applied to each individual frame, can be divided into two main steps: Static calibration: the first step is a static calibration of the arms, prior to each data collection, to define the reference model of the subject anthropometry. Concretely, the human subject was told to hold their arms full extended aligned with the trunk (fundamental standing position), while several frames are acquired. A distance vector among consecutive joints (shoulder-elbow and elbow-wrist) is calculated as the mean value taken over all these frames for both arms. It should be pointed out that this arm calibration is the basis for the joint-angle calculations in Section IV: all joints angles are defined as zero degrees at this calibration posture.

3 Pose correction: the basic problem is to find the closest 3 n configuration X = ( x1, x2,..., x n ) R 3 ( x 1,... x n R ) to the measurements that are observed over time, such that the distance between consecutive points (i.e., link lengths) remains constant. In line with this, we deal with the following optimization problem: n min X Xˆ (1) X Ω i = 1 where Ω is a certain subset of and is an appropriate matrix norm which measures goodness of fit. Here, we admit the Euclidean norm as measure of closeness. The goal is to minimize the objective function (1) by selecting a value of X that satisfies all equality quadratic constraints defined by: x i xi + 1 = di, i + 1 (2) where the left part is the Euclidean distance between two consecutive points and the right part is the link lengths in the reference model. The constrained minimization problem was solved with the OPTI toolbox that can solve this problem of optimizing a quadratic function of several variables subject to quadratic constraints. The comparison of the human skeletons obtained with the Kinect raw-data and those after the pose correction are illustrated in Fig. 2. Different poses are represented for a movement sequence involving both the right and the left arm. Fig. 2. Overlap of the human skeletons extracted from the Kinect and those after the constraint-based optimization at different frames (green points and black lines are original Kinect data; red points and blue lines are motion constrained filtered data. Green and red lines are the respective trajectories of the wrists). IV. KINEMATIC MAPPING One of the main issues in using motion capture data for training robots is to convert the 3D joint positions into joint angles relative to a robot model. In this context, the human skeleton is replaced by two 4 degree-of-freedom (DOF) robot arms of the same dimensions. Then, an inverse kinematics algorithm generates the corresponding joint angles of the robot for each pose. The problem is decomposed into a per-frame inverse kinematics algorithm, followed by motion filtering and interpolation. A. Inverse kinematics The filtered movement data is the input for the inverse kinematics module in which the human arms are modeled as two independent 4-dof serial chains consisting of a 3- DOF shoulder (rotations joints with intersecting axis) and a 1-DOF elbow joint. The implementation of the inverse kinematics follows some basic assumptions. First, the robot model was defined to match the anthropometric measures of the human subject, avoiding the retargeting problem (i.e., compensate for body differences). Second, the perturbations in the movement data caused by the movement of the subject s shoulder are ignored. Concretely, we consider that all joint positions are uniformly affected by the perturbations and the shoulders are at the origin of the reference system with fixed coordinate frames. Third, the inverse kinematics considers mechanical constraints on the joints, such as physical limits both on the range of joint motions (e.g., the elbow cannot invert the motion when fullstretched) and on the maximum joint velocities. Given the 3D positions of the shoulder, elbow and wrist, the inverse kinematics algorithm is simplified: two degrees of freedom completely describe the elbow when the position of the shoulder is known (the elbow lies on the surface of a sphere centered at the shoulder). Similarly, the wrist can only lie on the surface of a sphere centered at the elbow. Thus, the configuration of the arm is completely represented by four variables (the joint angles). Attention was devoted to avoid discontinuous jumps near ±180º associated with the use of inverse tangent functions. Additionally, the implemented algorithm includes a validation test since there may be motions where the robot s joints are not able to approximate the human pose in a reasonable way due to physical limitations. The proposed strategy to properly cope with the joint velocity limits is to slowing down the task-space trajectory whenever the limits are encountered. Thus, whenever the generated joint velocities violate the speed limits of the joint actuators, the trajectory is scaled in time by an appropriate constant that simultaneously assures tracking of the desired arm path and the fulfillment of the velocity constraints. B. Filtering and Interpolation The frame rate of the Kinect sensor and high frequency components in the movement data imposed a postprocessing stage to refine results. The exact procedure combines basic interpolation and smoothing techniques. On the one hand, the joint-angle trajectories are filtered using a moving average algorithm to smooth out short-term fluc-

4 tuations based on predefined trail onset and termination times. On the other hand, the strategy adopted to provide a more detailed description of the action performed by the human subject is to use spline interpolation over the set of observations to satisfy the requirements of differentiability. To evaluate the different steps of post-processing, we use a measure based on jerk, the third time derivative of position, to quantify smoothness at the level of the joint-angles trajectories. Concretely, the particular jerk metric used to quantify movement smoothness is the integrated squared jerk [12] defined by: t η = 2 ( t) dt (3) isj t 1 x A comparison of movement smoothness measures among the original signal (after pose correction), the moving average filtered signal, the cubic spline interpolation and the fifth-order spline interpolation was performed (Fig. 3). The exact procedure to be followed depends on the ultimate goal. Anyway, the previous considerations may be of importance in determining what strategies are appropriate to the problem in hand. Fig. 4. Variability of human movements in the task-space for the execution of a circular path repeated many times. Fig. 5. Variability of human movements in the joint-space for the execution of a circular path repeated many times. Fig. 3. Comparison of the smoothness measure for different motion postprocessing methods applied on the joint-angle trajectories (ordinate is plotted in a logarithmic scale). V. GESTURES IMITATION IN A ROBOT Several real-time movements executed by a human subject were captured using the Kinect sensor to provide validation for our algorithms. Two different movements were chosen to illustrate the results: a rhythmic motion repeated many times and a discrete sequence of upper-limb movements. In the first experiment, the human subject is asked to repeat a circular path trying to keep, as much as possible, a constant speed across all trials. Fig. 4 and Fig. 5 show the variability always present in human movements, both in task- and joint-spaces. Since the details vary, it seems necessary to consolidate the demonstrated movements having in mind the desired final result (i.e., the extent to which the motor goal is reached). The second experiment consists of a gesture imitation task using the two arms in different configurations around the T-pose. Fig. 6 compares the positions of the right and left wrists as seen by the filtered data and the robot simulation. The consistency between the two curves suggests the efficacy of the human motion reconstruction algorithm proposed. VI. CONCLUSIONS In this paper, we have described and demonstrated the potential of the Kinect sensor for gestures imitation of an upper-body robot from demonstrations of a human teacher. The implementation of the proposed ideas on the 4-DOF robot model shows that human-demonstrated gestures are well replicated by the robot. In this context, the approach is useful for providing a natural and intuitive interface for a user to teach complex movements to a robot. The main goal is to create real data sets that, if combined with other, can be later used for learning a compact representation of the task. In this context, they will assist in developing learning techniques for manipulation/locomotion behaviors based on examples of human demonstrations.

5 Fig. 6. Comparison of the motion capture data (left) with the gestures replicated by the robot (the end-effector path is represented in both cases). ACKNOWLEDGMENT This work is partially funded by FEDER through the Operational Program Competitiveness Factors - COMPETE and by National Funds through FCT - Foundation for Science and Technology in the context of the project FCOMP FEDER (FCT reference Pest-C/EEI/UI0127/ 2011). Zhenli Lu is supported by FCT under contract CIENCIA 2007 (Post Doctoral Research Positions for the National Scientific and Technological System). REFERENCES [1] Billard, A., Callinon, S., Dillmann, R., Schaal, S.: Robot Programming by Demonstration. In: Siciliano, B., Khatib, O. (eds.) Handbook of Robotics, Springer, New York, NY, USA, (2008)

6 [2] Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A Survey of Robot Learning from Demonstration. Robotics and Autonomous Systems, 57(5): (2009) [3] Dasgupta, A., Nakamura, Y.: Making Feasible Walking Motion of Humanoid Robots from Human Motion Capture Data. In: IEEE International Conference on Robotics and Automa-tion, pp (1999) [4] Elgammal, A., Lee, C-S: Tracking People on a Torus. IEEE Transactions on Pattern Anal-ysis and Machine Intelligence, 31(3): (2009) [5] Inamura, T., Toshima, I., Tanie, H., Nakamura, Y: Embodied Symbol Emergence Based on Mimesis Theory. International Journal of Robotics Research, 23(4-5): (2004) [6] Kulic, D., Takano, J.W., Nakamura, Y.: Incremental Learning, Clustering and Hierarchy Formation of Whole Body Motion Patterns using Adaptive Hidden Markov Chains. Inter-national Journal of Robotics Research, 27(7): (2008) [7] Shon, A.P., Grochow, K., Hertzmann, A., Rao. R.P.: Learning Shared Latent Structure for Image Synthesis and Robotic Imitation. In: Weiss, Y., Schlkopf, B., Platt, J.C. (eds.) Ad-vances in Neural Information Processing Systems, MIT Press, Cambridge, MA (2005) [8] Shotton, J., Fitzgibbon, A.W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.; Real-time Human Pose Recognition in Parts from Single Depth Images. In: IEEE Computer Vision and Pattern Recognition, Colorado Springs, USA (2011) [9] Khoshelham, K., Elberink, S.O.: Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications. Sensors, 12(2): (2012) [10] Smisek, J., Jancosek, M., Pajdla, T.: 3D with Kinect. In: International Conference on Computer Vision Workshops, pp , Barcelona, Spain (2011) [11] Obdržálek, S., Kurillo, G., Ofli, F., Bajcsy, R., Seto, E., Jimison, H., Pavel, M.: Accuracy and Robustness of Kinect Pose Estimation in the Context of Coaching of Elderly Population. In: International Conference of the IEEE Engineering in Medicine and Biology Society, pp , California, USA (2012) [12] Platz, T., Denzler, P., Kaden, B., & Mauritz, K.-H: Motor Learning After Recovery from Hemiparesis. Neuropsychologia, 32, (1994)