ABSTRACT Using Artificial Neural Networks for Prediction Of Dynamic Human Motion Researchers in robotics and other human-related fields have been studying human motion behaviors to understand and mimic them in humanoid motion prediction, obstacle avoidance, and ergonomic studies. Human motion, however, is not an easy system or kinematic to study when it includes highly complex relationships between factor such as human anthropometry and speed and the output motion profile for human degrees of freedom (DOF) involved in the task to be accomplished. Artificial neural network (ANN) is a method that has been introduced to analyze motion prediction problems because of its power in studying high-dimensional problems and predicting future system behaviors. In this study we used a general regression neural network (GRNN) to predict the human walking forward task as an example of ANN s ability to predict human performance. The results showed that the GRNN was able to predict the motion realistically, accurately, and by a fraction of a second. This study shows that ANN has great potential to be widely used in task-based prediction of dynamic human motion. The novelty of this work is demonstrated by using ANN to predict human performance by studying motion prediction as an example. This will lead to an understanding of what drives human task performance. 1. INTRODUCTION Digital human modeling is an extensive field that contributes in many studies that mimic what is done by human beings. Currently, digital humans and humanoids can do a lot of intelligent tasks such as moving, touching, and visually interacting. Researchers, however, are still studying 1
and analyzing many unique and complicated issues related to human motion and posture prediction. Questions like why we sit and stand as we do need to be analyzed further to fully understand human decisions about motion and posture. Obstacle avoidance and fear response incorporate a combination between visual, muscular, and mechanical efforts. Again, how can we study all these human mysteries? Many techniques have been introduced in predictive dynamics (PD) and motion prediction, such as physiological constraints that shape human motion (Khatib et al., 2004). In this research, these characteristics are mapped for robotic control by incorporating potentials into a prioritized task-level control framework. They investigated the muscular effort required for reaching tasks using this approach and then constructing a basis of human motion strategies. Liang et al. (2010) studied human movement using motion capture systems to analyze and determine the effort level of each key joint using inverse kinematics. Then the physical fatigue level of each joint is further analyzed by integrating a fatigue and recovery model on the basis of physical task parameters. His method has demonstrated the function of physical fatigue for a given population and its usefulness for worker selection. The scholars have provided many methods to study deeply the heart of human behavior. One of the most interesting methods that has been introduced is the artificial neural network (ANN). It has been used in control systems for moving parts as well as in humanoids movement. In addition, it is used in gait and motion analysis because this type of analysis includes complex relationships between each angle and its corresponding control points as well as motions of the other angles. Feed-forward ANN was used to manipulate an electro-myogram (EMG) signal and 2
joint motion in predicting a joint s stiffness control strategy during a specific contact task (Kang et al., 2007). The self-organizing map ANN was used as a control scheme that is able to predict human motion in a human-robotic environment (Najmaei et al., 2011). The ANN receives information about the human footprint through a sensory system; this represents ANN s input. Then it was able to provide output that is represented by coordinates (x and y) for the human s expected next steps to facilitate robot movement in such an environment without any interference or collision with the human. In addition, ANN, because of its calculations power, was used for feature extraction of these gait control points and corresponding angles. Dejnabadi et al. (2008) applied ANN to consider the nonlinear relationship between control parameters and angles. The error or difference between the trained ANN results and the real scores was less than 5.3%, which is an indication of how helpful ANN could be in finding these complex relationships without the need for other mysterious formulations or time-consuming techniques. In most parts of human motion prediction, scholars have studied it to help solve other issues related to motion but not motion itself. No study, to the best of the author s knowledge, has worked on predicting all body joints over motion profile or complete human motion prediction on a task based manner. The literature provides a potential use of ANN in human motion prediction because of its previously mentioned strengths, and some of studies have already incorporated some human degrees of freedom (DOF) in their designs. We found, however, that no one has predicted a complete human model DOF as an output for system design. Tracking motion profiles means hundreds of outputs to represent all DOF during any task. Motion profiles are affected, and changed by changing some or all of the task inputs. Hence, task 3
parameters (inputs) also are challenging issues that need to be chosen and specified carefully and that impact the motion profile. The literature has never touched this issue or related ANN directly to predicting human motion over a full task. This paper demonstrates the walking forward task as an example for motion prediction using the general regression neural network (GRNN) type of ANN. In the method section, we discuss the network architecture and the human model that we use in this study. The results are then provided in the next section, which provides a testing for three cases. The conclusion is then provided in the last section 2. METHOD Since this study is about using the ANN in human motion prediction, we applied the study on a digital human model. In addition, ANN has many types, and each has its own architecture and properties. For the network training, we briefly describe the source of the training cases as well. 2.1 Human Model The human model that we use in this study is Santos, which is virtual human software built by the Virtual Soldier Research (VSR) program at The University of Iowa (Figure 1). This human model mimics the human body by having 55 DOF, which represent all body joints, and all the links that connect these DOF (Abdel-Malek et al., 2006). Santos s skeleton is modeled to move as a series of links where each pair of links is connected by one or more revolute joints. Each joint could contribute in one or more DOF, and, depending on the task, some or all joint revolutions will produce the motion of the skeleton as one composite. In Figure 1, Santos s joints are located where different links join. At the joint location, there are one or more DOF, depending on the anatomical structure of that joint in the natural human. The green cylinders 4
represent the different DOF provided by various joints. The model links are indicated by the black arrows and named (L1 to L28). During this study, we used Santos as an application to study the ability of ANN to predict motion. 2.2 General Regression ANN (GRNN) The GRNN (Figure 2) is a type of ANN that is trained fast and provides accurate results because it works as a curve fitting for the network input parameters. It uses a radial basis function (Figure 3) as an activation function for its hidden layer and the most accurate type of ANNs at on-grid training points while it smoothes out the results for off-grid points without any strange results like other types of ANN. The GRNN has two hidden layers; one is the input hidden layer (1 st layer) that extracts the input features and passes its output to the next layer (2 nd hidden layer), which provides the outputs based on the input from the first layer. The number of hidden neurons at the first layer is equal to the number of training cases, while the second layer is equal to the number of outputs. The following discussion explains the working strategy of the GRNN mathematically. The GRNN is a special type of radial basis neural network (RBNN), which is described by Wasserman et al. (1993). The GRNN has an output that depends on the radial distance of the input from the function s center, shown in Equation 1. The equation represents the mathematical formula for the radial function that is shown in Figure 3 (Buhmann et al., 2003). After the input features (X) enter the network, each value goes to all hidden neurons. Inside the neuron, the difference between each input and its corresponding weight (Wi) is taken and is passed to the radial basis function. Then, the radial function output (between 0 and 1) enters 5
the next hidden layer, where it multiplies by the second hidden layer neuron (Wj) to provide the output of the network. The network s first layer is designed to have a number of neurons equal to the number of training cases where each neuron has a vector of weights with values equal to the input vector for a specific training case. So, the network can handle the input features by saving all training cases as weights for the first hidden layer. Figure 4 illustrates the underlying details of a neuron in the first hidden layer, which has a vector of neurons. The network itself has R number of inputs and Q number of training cases. Inside the neuron, as it is shown in Figure 4, the weights are represented by a vector of 1xR, where it receives lines from all inputs (size of R). During the training process, the neuron s weight elements are set to be equal to the training input vector; Equation 2 shows the first layer s neuron elements. In Equation 2, each row represents one neuron; we have Q rows to represents all training cases. In Equation 3, the inputs are subtracted from the weight vector to give Z. Then, the result Z is multiplied by the bias (B1), which is responsible for network sensitivity, using dot product, yielding the net value a, shown in Equation 4. The result (a), which is a vector, represents the sum of all distances between all input features and their assigned weights, and it passes through radial function (h), which is shown in Equation 5, to produce values between 0 and 1. Outputs of the first hidden layer (h1 to hq) then enter the second hidden layer, where a weighted-sum multiplication occurs. Equation 6 shows the second layer s weight matrix (Wj), which has size equal to the network outputs (K). The output vector (Y) is obtained from Equation 7 and is a result of the multiplication of (h) by (Wj) and normalized by dividing by the sum of (h) elements. In the sum, the network provides proper outputs depending on the weighted sum of the inputs that it trained on. 6
2.3 Predictive dynamic algorithm The most important factor in deciding whether to use ANN to predict such a system behavior is having reliable and sufficient training cases for the network. The VSR program developed a PD algorithm for motion prediction. This code depends on the mathematical formulation of an optimization approach for motion prediction. They validated this algorithm performance on Santos by comparing its results to the motion capture system output for normal humans. The walking task, which is the task we study in this paper, is one of the tasks that have been validated for this algorithm. Hence, the training cases for this study were obtained from a reliable source. Using ANN in motion prediction depending on a PD algorithm is an introductory step to using it with real data and tasks provided directly from motion capture (MC) systems. Eventually we will use MC as a source of training for the ANN after defining the task and its variables. We depend on PD in this study because it is easier to collect the training cases. In addition, PD is more cost-effective because MC needs many subjects to provide the proper training cases. So, we preferred to do our study using PD to test the effectiveness of using ANN in motion prediction before incorporating MC. 3. RESULTS In this study, we used GRNN to predict the walking forward task on Santos as an example of its ability to handle the complexity of human motion. The variables used to define the input task include: 1) backpack weight, 2) the four lower-body link lengths (spine to hip, hip to knee, knee to ankle, and ankle to foot), 3) motion velocity, and 4) lower-body joint ROMs. Different ROMs include flexion extension for the hip, knee, and ankle and in both sides (upper and lower limits). 7
Those specific joint ROMs are used because they are the most effective joints during the walking task, and changing their limits will change the walking profile. In addition, in the walking task, the flexion-extension motions are the most significant ROMs in this motion; most ROMs take part in the motion profile. The output of the network was defined to be Santos s joint values during motion profile (joint splines) and torque. As we described in the literature review, Santos s skeleton has 55 DOF, and during motion they are defined by the splines (control points) that define each value of those DOF over the motion. Thus, the results include splines for all 55 DOF. Each of those splines has 6 control points. In total we have 330 output lines for all Santos s skeleton joints. Moreover, we included joint torques for the lower joints because they are highly articulated during the walking task. Hence, we added 60 more output lines corresponding to torque values over the timeframe of the task. Those 60 are for the 6 lower joints that we mentioned above, where there are 10 timeframes for each one of them over the motion profile. To summarize, the task has 12 inputs and 390 outputs in total, and GRNN is used to figure out the relationships between those inputs and provides the proper outputs corresponding to them. The training process was completed using 52 training cases that represent different combinations for the inputs. The training cases were collected from the PD algorithm designed by the VSR program. This algorithm also was validated using motion capture data to make sure it was a real motion. Table 1 shows the inputs and their values that were used to train the network. The 3 different values for each input are combined with the other inputs to provide the 52 training cases that we used in this study. 8
The results of the GRNN are compared visually and statistically with that from the PD algorithm. They were promising and accurate subjectively and objectively. The testing case s inputs are represented in Table 2. In terms of visual results, Santos s motion produced by GRNN looks normal and similar to that which comes from the PD algorithm. Figure 5 represents static visual results for the testing cases. From this figure, we could see how the results from the GRNN are very close to what the PD provides. At the same timeframe in the motion profile, the joint and link positions in GRNN and PD match well. The back also bends in a similar manner for GRNN and PD for all cases, which indicated the accuracy that the GRNN could provide even with this huge number of outputs (390 outputs). Statistically, the plots of adjusted R-square value were drawn for PD and GRNN joint torque values for accuracy comparison purposes. Figure 6 shows adjusted R-square for torque values between PD and GRNN results. These plots include the 10 timeframe values for each of the six joints of interest that we mentioned above. The results show that there is a good and accepted match for various cases. In addition, the GRNN needs a fraction of a second to provide the output, while the PD algorithm needs between 5 and 30 minutes, depending on the input conditions, to optimize and provide the same output. Each of the collected training cases, which are collected using the same computer, needed the same above time range to be completed by the PD. The GRNN training, however, took a fraction of a second to be completed, which shows another advantage of using the GRNN in motion prediction in terms of speed. 9
4. CONCLUSION In this study, we used a GRNN to predict the human walking forward task as an example of ANN s ability to predict human performance. The network is trained to handle a human motion prediction task. We applied this study on Santos, which is a 55-DOF virtual human model built by the VSR program. The 12 inputs for the network include step size, backpack weight, link lengths, and some key joint range of motions. The 390 outputs include 330 for 55 DOF (6 control points for each), and 60 for 6 joint torques (10 control points for each). To learn the network, we used 52 training cases obtained from a validated mathematical formulation for walking task prediction on Santos done by the same group. The algorithm needs hours to finish running all these cases, and there are infinite combinations of inputs, which indicates that time is a problem in this algorithm. After training, the results show that the GRNN was able to predict Santos s motion realistically, accurately, and by a fraction of a second, even for this large number of inputs and outputs and off-the-grid training points. This study shows that ANN has great potential to be widely used in task-based motion prediction of humanoid work and motion. The novelty of this work would be concluded by using ANN to predict human performance, which is complicated for tracking, by taking motion prediction as an example. It is a promising method that we would use in understanding task-based motion prediction as well as work environments. The challenges for using ANN in motion prediction, which need to be solved in future work, include: 1) professional and automatic task definition, 2) incorporating all joint torques in the outputs, 3) increasing the accuracy of the predicted torques, 4) identifying the optimal number of training cases for a task, and 5) studying more complicated tasks like rolling. 10
Figures list Figure 1: The Virtual Human Santos Figure 2: General regression neural network architecture Figure 3 Radial function Training values Value 1 Value 2 Value 3 Velocity (m/s) 0.8 -- 1.6 Backpack weight (N) 0 175 350 Link1 (Spine to Hip) (cm) 7.8 8.8 9.8 Link2 (Hip to Knee) (cm) 43.5 44.5 45.6 Link3 (Knee to Ankle) (cm) 39.5 42.4 45.4 Link4 (Ankle to Football) (cm) 11.3 11.7 12.1 Joint1 (Hip)- lower limit (degrees) -123.3-105 -90 Joint1 (Hip)- upper limit (degrees) 8.7 5 2 Joint2 (Knee)- lower limit (degrees) 5 10 20 Joint2 (Knee)- upper limit (degrees) 149.7 130 110 Joint3 (Ankle)- lower limit (degrees) 7.3 15 20 Joint3 (Ankle)- upper limit 71.6 60 50 Figure 4: First hidden layer neuron Training values Case 1 Case 2 Case 3 Velocity (m/s) 1.4 0.9 1.1 Backpack weight (N) 220 63 315 Link1 (Spine to Hip) (cm) 9 8 9.6 Link2 (Hip to Knee) (cm) 44 43 45.4 Link3 (Knee to Ankle) (cm) 41.4 45 43 Link4 (Ankle to Football) (cm) 12 11.3 11.6 Joint1 (Hip)- lower limit (degrees) -98.6-111 -90 Joint1 (Hip)- upper limit (degrees) 2.2 6.5 4.5 Joint2 (Knee)- lower limit 8 16 19 (degrees) Joint2 (Knee)- upper limit 146 127.2 112.3 (degrees) Joint3 (Ankle)- lower limit 12 7 16 (degrees) Joint3 (Ankle)- upper limit 57 70 53 Table 1: Input variables that used in walking task.each one of them has its unit of measurement between brackets after the variable name is mentioned Table 2: Input variables for the three testing points. 11
Figure 6 Adjusted R-square for six torque values (hip, knee, and ankle joints) for the three testing cases. 12
REFERENCES [1] Buhmann, M.D., Radial Basis Functions: Theory and Implementations. Cambridge University Press, ISBN 978-0-521-63338-3, 2003. [2] Khatib, O., Sentis, O.L., Park, J.H., and Warren, J. Whole body dynamic behavior and control of human-like robots, Int. J. Human. Robot. 1(1), 29 43, 2004. [3] Abdel-Malek, K., Yang, J., Marler, R., Beck, S., Mathai, A., Zhou, X., Patrick, A., and Arora, J. Towards a new generation of virtual humans: Santos, International Journal of Human Factors Modeling and Simulation 1(1), 2 39, 2006. [4] Ma, L., Chablat, D., Bennis, F., Hu, B., and Zhang, W. Integrating digital human modeling into virtual environment for ergonomic oriented design. Paper presented at the World Conference on Innovative Virtual Reality, France, 2010. [5] Kang, B., Kim, B., Park, S., and Kim, H. Modeling of artificial neural network for the prediction of the multi-joint stiffness in dynamic condition, in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., pp. 1840 1845, 2007. [6] Najmaei, N., and Kermani, M.R. Applications of artificial intelligence in safe human robot interactions. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics 41(2), 2011. [7] Hooman, D., Jolles, B.M., and Kamiar, A. A new approach for quantitative analysis of interjoint coordination during gait. IEEE Transactions on Biomedical Engineering 55(2), 755-764, 2008. [8] Wasserman, P.D. Advanced Methods in Neural Computing, 155 161. Van Nostrand Reinhold, New York, USA, 1993. 13