Learning Humanoid Motion Dynamics through Sensory-Motor Mapping in Reduced Dimensional Spaces

Transcription

1 Learning Humanoid Motion Dynamics through Sensory-Motor Mapping in Reduced Dimensional Spaces Rawichote Chalodhorn, David B. Grimes, Gabriel Y. Maganis and, Rajesh P. N. Rao Neural Systems Laboratory Department of Computer Science and Engineering University of Washington Seattle, WA U.S.A. Abstract Optimization of robot dynamics for a given human motion is an intuitive way to approach the problem of learning complex human behavior by imitation. In this paper, we propose a methodology based on a learning approach that performs optimization of humanoid dynamics in a lowdimensional subspace. We compactly represent the kinematic information of humanoid motion in a low dimensional subspace. Motor commands in the low dimensional subspace are mapped to the expected sensory feedback. We select optimal motor commands based on sensory-motor mapping that also satisfy our kinematic constraints. Finally, we obtain a set of novel postures that result in superior motion dynamics compared to the initial motion. We demonstrate results of the optimized motion on both a dynamics simulator and a real humanoid robot. Minoru Asada JST ERATO Asada Synergistic Intelligence Project Department of Adaptive Machine Systems Graduate School of Engineering, Osaka University Yamadaoka -, Suita, Osaka , Japan asada@ams.eng.osaka-u.ac.jp behaviors. A framework that is based on human designed behaviors may lack the ability to develop behaviors through embodiment [8]. The mimesis theory [9] proposed action acquisition and action symbol generation while considering the embodiment concept. However, the mimesis framework [9] does not provide a dynamics compensation for biped locomotion that is applicable in real-time. Motion tracking Kinematics mapping Low dimensional subspace representation Index Terms- biped locomotion, dimensionality reduction, humanoid robot, optimization. I. INTRODUCTION Learning of a complex task Robot dynamics compensation Learning by imitation is one of the most interesting approaches to make a humanoid perform a complex human task. Even when a corresponding motion is kinematically feasible, performing the motion with stable robot dynamics is a challenging research problem. Optimization of robot dynamics over a given motion is an appealing approach to the problem yet it suffers from two problems. First, model-based approaches using zero-moment point (ZMP) [], [], or the inverted pendulum model [3] plan desired trajectories and control their bipeds to follow them. In order to achieve a stable walking gait, these methods require highly accurate model for robot dynamics and the environment. Second, learning approaches such as reinforcement learning [4] provide an optimal solution yet can adapt to the environmental change. However, such methods are not directly applicable to humanoid robots because of the curse of dimensionality problem involving the high dimensional search in joint space. Okada, Tatani and Nakamura [5] were the first to apply non-linear principal components analysis (NLPCA) [6] to human and humanoid robot motion data, though for dimensionality reduction only. A number of imitation frameworks have also been proposed. A nonlinear dynamical system [7] was carefully designed to produce primitive Fig.. A framework for learning human behavior by imitation through sensory-motor mapping in reduced dimensional spaces. A motion segmentation framework [], which uses dimensionality reduction and segmentation of motion data in the reduced dimensional space, also does not show any dynamics compensation. An algorithm that segments humanoid motion data automatically in reduced dimensional space [] also does not demonstrate how to regenerate a dynamically stable motion explicitly. In this paper we approach the problem of imitation of humanoid motion including body dynamics. The framework on which our method is based is shown in Fig.. First, an optical motion capture system transforms Cartesian position of markers attached to the human body to joint angles based on kinematic relationships between the human and robot bodies. In order to be able to perform optimization of the robot dynamics efficiently, we employ dimensionality reduction to represent posture information in compact low-dimensional

2 subspaces. Periodic motion patterns are generalized as closed curves []. Subsequently, we can re-generate motion on the robot using an inverse mapping from the reduced space back to the original joint space. Sensory feedback data are recorded from the robot during motion. Then a causal relationship between actions in the low dimensional posture space and the expected sensory feedback are learned by a nonlinear regression model. A sensory-motor learning algorithm allows the representation of such a non-linear relationship between the sensory feedback and motor commands. Finally, actions that imitate input postures while maintaining imposed criteria such as dynamic stability of the body are selected. we are using a rhythmic walking gait generator [4] for our initial training set. The idea is to use this motion as a seed motion. A reduced set of basis vectors is obtained corresponding to the m ψlargest eigenvalues of the covariance of Z ψafter subtracting the mean of each dimension. The result can be thought of as two linear operators C and - C which map from the high to low, and low to high dimensional spaces respectively. An example of such a space, along with corresponding postures is shown in Fig.. B. Action subspace embedding II. SENSORY-MOTOR MODELING FRAMEWORK A. Reduced posture dimensionality o φ Z Y X Fig. 3. Embedded action space of a humanoid walking gait. Training data points in the reduced posture space (shown in blue-dots) are converted to a cylindrical coordinate frame relative on the coordinate frame x,y,z. The points are then represented by a function of the angle φ, which forms an embedded action space (shown in red-solid-curve). This action space represents a single gait cycle. Fig.. Posture subspace and example poses. A three dimensional space represents the posture of the Fujitsu HOAP robot. PCA was used to reduce dimensionality from robot posture space to the three dimensional space as shown. Blue points along a loop represent different robot postures during a single walking cycle. Red points mark various example poses as shown in the numbered images. The first two postures are intermediate postures between an initial stable standing pose and a point along the periodic gait loop represented by postures three through eight. Particular classes of motion such as walking, kicking, or reaching for an object are intrinsically low dimensional. More precisely, the variance of posture over time and different styles/instances of an action is largely distributed in a subspace with far fewer directions of variance. Thus we apply the well known method of principal components analysis (PCA) to parameterize the low dimensional subspace X. Research has revealed that nonlinear methods [], [3] can also be used to reduce full posture space Z. For simplicity we use the standard linear PCA method in this paper. We construct the reduced dimensionality space or feature L space using a set of initial training examples Z = z... z where L is number of data sequence in a data set. Tentatively High-level control of a humanoid robot can be seen as selecting a desired angle for each joint servo. As discussed previously, complex operations in the space of all joint angles taken together are often intractable. Again we leverage the redundancy of the full posture space and use X to constrain target postures. Any desired posture (also referred to as an action) can be represented by a point a X. Further, we show that space of desired postures can be represented more compactly by a non-linear manifold embedded in X. Fig. 3 illustrates a fixed periodic movement such as walking represented by a loop (parameterized by time) in X. In the general case we consider a non-linear manifold representing the space A X of actions. Non-linear parameterization of the action space of desired postures allows for further reducing the dimensionality in the optimization process in the model-predictive control algorithm that we use in Section III. The methodology presented in this paper embeds a one dimensional representation of the original motion in a three dimensional feature posture space and uses it for constructing a constrained search space in the optimization process. Using

3 the feature representation of the set of initial training examples i i x = Cz we first convert each point into a cylindrical coordinate frame. This is done by establishing a coordinate frame represented by three basis directions x,y,z in the feature space. The zero point of the coordinate frame is the empirical mean of x i, denoted µ. Thus we recenter the data around this new zero point and denote the resulting data x ˆi. We then compute the principal axis of rotation zˆ accordingly: ˆ i ˆ i+ ˆ = (x x ) i z. () i i+ (x ˆ x ˆ ) i i Next, x is chosen to align with the maximal variance of x in a plane orthogonal to z. Finally, y is specified as orthogonal to x and z. The final embedded training data is obtained by cylindrical conversion to( φ,r,h)where r is the radial distance, h the height above the x y ψplane, φ is the angle in the x y plane. The angle φ can be also denoted as the motion phase angle of the action subspace embedding. Given the loop topology of the latent training points, one can then parameterize r ψand h ψas a function of φ. The embedded action space is represented by a learned approximation of the function [ r,h] = g(φ) () where φ π. Approximation of this function is performed by using a radial basis function (RBF) network (described in Fig. 3). Prior research has studied this approximation process in depth [], however for reasons of fast convergence and more importantly robustness to local minima here we utilized a standard RBF network. C. Sensory-motor prediction Posture command Humanoid Robot Time-Delay RBF Network Learning Algorithm Gyroscope signal - Error Fig. 4. Sensory-motor stability prediction module. Motion stability (as measured by a gyroscope in the torso) is predicted based on the input whole body posture command. The predictor network is trained by comparing the predicted gyroscope sensor reading vs. the actual sensor reading from the robot Fig. 5. Gyroscope signal prediction. A second-order radial basis function network is able to accurately predict gyroscope signals at the next time step. The plots from top to bottom represent individual gyroscope signals x, y and z during many periods of the walking gait. A part of our proposed framework is learning to predict future sensory inputs and utilizes such a predictor in optimal action selection. A sensory-motor mapping predicts the future state of the robot, denoted by s t+. In general the state space S= Z Ρ is the Cartesian product of the high dimensional posture space Z and the space of other percepts Ρ. Other percepts could include a torso gyroscope, accelerometer, and foot pressure sensors as well as information from camera images. To summarize, one can see the prediction as capturing the function F : S A a S. For the purposes of this work we assume that F ψis deterministic. Often a perceptual state s t cannot fully represent the true real world state of the robot. To increase the accuracy of the sensory-motor prediction we allow for a higher order mapping based on a history of perceptual states and actions. Such a mapping can be written as an n -th order Markovian function: s t = F (s t-n,...,s t-,a t-n,...,a t-). (3) Rather than attempting to model F explicitly, we propose learning F through direct sensory experience. Our approach allows for the function to be of arbitrary complexity by utilizing a form of non-parametric function approximation. Given the highly non-linear nature typical in sensory-motor mappings of real systems we utilize kernels with local support such as exponential-quadratic units. When combined with a linear output network, our mapping takes the form of a radial basis function (RBF) approximator. A function F' : α a β is approximated: K T ( k ) β = w exp (α µ ) (α µ ), (4) k Gyroscope signal of X-axis Gyroscope signal of Y-axis Gyroscope signal of Z-axis k k k actual signal predicted signal actual signal predicted signal actual signal predicted signal where K represents the number of kernels, µ k and Σk are the mean and covariance of the k-th kernel respectively. Finally,

4 the output weight vector wk scales the output of each kernel appropriately. Sensory-motor prediction uses an RBF network with α = [ s t, s t-,...,s t-n-,a t,a t-,...,a t-n-] as input andβ = s t+ as output. For convenience one can instead view the RBF as a time delay network [5] for which the input simplifies to α = [ s t,a t]. The previous state and action inputs are implicitly remembered by the network using recurrent feedback connections. In this paper we use a second-order (n = ) RBF network with the state vector equal to the three-dimensional gyroscope signal ( st ω t). As discussed in the previous section, an action represents the phase angle, radius and height of the data in latent posture space ( at χ X ). This specific form of the sensory-motor predictor used in this paper is detailed in Fig. 4. Predicted gyroscope data versus actual gyroscope data during a motion test sequence is shown in Fig 5. ωmin III. MOTION OPTIMIZATION AND RELEARNING Optimization Algorithm ω min = minimum gyroscope signal ω a = actual gyroscope signal ω p = predicted gyroscope signal Controller χ χ Model Predictor ω p ω a Humanoid Robot Gyroscope signal χ = posture command χ = tentative posture command Fig. 6. Model predictive controller for optimizing posture stability. The optimization algorithm and the sensory-motor model predictor produce the action ( at χ X) which is used for posture control of the humanoid robot. The resulting gyroscope signal is feed back to the predictor for retraining. The optimization algorithm utilizes a predicted gyroscope signal ω in order to p optimize actions for posture stability. Our proposed method is diagramed in Fig. 6. The algorithm we present in this section utilizes optimization and model prediction in concert to select optimal actions and control the humanoid robot in a closed-loop feedback scenario. For our framework one may express the desired sensory states that the robot should attain through an objective function Γ( s ). Our algorithm then selects actions * * t,..., T a a such that predicted future states st,..., st will be optimal with respect to Γ( s ) : ( F ) * a = arg min Γ ( s,..., s, a,..., a ). (5) t t t-n t t-n a t The objective function used in this paper is a measure of torso stability as defined by the following function of gyroscope signals: (ω) = λxωx λyωy λzωz Γ + +, (6) where ω x,ω y,ω z refer to gyroscope signals in the real world coordinate system x, y, z axes, respectively. The constants λ x, λ y,λz allow for weighting rotation in each axis differently, and are discussed further in Section IV-A. The objective function of (6) thus provides a measure of stability of the posture during motion. For this case a second-order predictive function F the optimization problem can be restated as follows: ( F ) χ * t = arg min Γ (ω t, ω t-, χ t,χ t- ) χ S t φs = r s hs φ t-< φs φt- εφ (6) S (7) + (8) ra - εr rs r a+ εr (9) ha - εh hs h a+ εh () < ε <π () φ [ r,h] = g(φ ) () a a s The search space S is defined by multi-dimensional ranges φ s, r s and hs in the cylindrical coordinate system x,y,z described in section II. The phase motion command search range φ s begins after the position of the phase motion command at the previous time step φ t -. The radius search range r s begins from a point in the action subspace embedding A that is defined by () and () in both positive and negative direction from r a along r with respect to x,y,z for the distance εr >. The search range h s is defined in the same manner of r s according to h a and ε h. One may arbitrarily choose the parameters ε φ, ε r and ε h. However, ε r and ε h implicitly enforce similarity between the optimized motion pattern and the initial motion pattern. The search space S for an optimal move χ * t is shown in Fig. 7. From (6), one must note that selected actions will only truly be optimal in the case that the sensory-motor predictor is accurate. Since at the first iteration of learning of the predictive model in (3), the sensory-motor data is limited to the initial data set then the initially prediction may yield significant errors. In order to achieve higher accuracy of the model predictor, we periodically re-train the prediction model

5 based on new posture commands that is generated from the optimization algorithm and its sensory data feedback. 8) Iteratively perform steps 4 through 7 repeatedly to improve the model predictor, and achieve increasingly optimal motion. B. Gait optimization results - Gyroscope signal of X-axis Original : RMS =.336 Optimized : RMS = Gyroscope signal of Y-axis Original : RMS =.459 Optimized : RMS = Fig. 7. Optimization result of a walking motion pattern in a low-dimensional subspace based on an action subspace embedding constraint. Gyroscope signal of Z-axis Original : RMS =.3795 Optimized : RMS =.533 IV. EXPERIMENTAL RESULTS In order to obtain a dynamically stable optimized walking gait, we performed multiple learning iterations in a commercially aviable robot simulator [6]. Subsequently we tested the optimized walking gait on the Fujitsu HOAP robot. A. Methodology We briefly review our motion generation methodology: ) Use PCA to represent the initial training data from the walking gait in reduced 3D space. ) Employ the non-linear embedding algorithm to allow for modifying the gait by selecting different actions at each time t. 3) Perform inverse mapping of actions back to the original joint space. Execute a series of motor control commands in the Webots HOAP simulation [6] model and record sensory feedback (shown in Fig. 7). 4) Learn a sensory-motor predictor as described in Section II-C where the state variable in our case are gyroscope signal of each axis and the action variables are φ,r and h in the low-dimensional subspace. 5) Implement the model predictive controller framework as described in Fig. 6. The controller makes use of the sensory-motor predictor to plan optimal action based on Equation 6. 6) Execute computed actions and record gyroscope feedback. 7) Perform learning update of the predictor module (step 4) Fig. 8. Comparison of gyroscope signals from initial and optimized walk The plots from top to bottom show the gyroscope signals for the axes X,Y, and Z recorded during initial and optimized walking motions. Root mean squared (RMS) of the gyroscope readings are also indicated in the plot legends. Notice that all of the RMS values are significantly reduced for the optimized motion. After three iterations of sensory-motor prediction learning, an improved dynamically balance walking gait is obtained. The trajectory of the optimized walking gait in the low dimensional subspace is shown in Fig. 7. The new trajectory has a similar shape to the initial one however it has a larger magnitude and offset of location from the initial pattern. After remapping this trajectory back to the high dimensional space, we tested the optimized motion pattern with the simulator and the real robot. The gyroscope reading of the new walking pattern is shown in Fig. 8. According to the convention in our OpenGL based simulator, the y-axis points in the vertical direction, the x-axis points towards the left side body, and the z-axis points forward out of the body. The root mean square (RMS) values of the optimized walking gait according to X, Y and Z axes are.5,.5 and.533 respectively, while the gyroscope reading of the original walking gait are.336, 459 and The RMS values from the optimized walking gait are significantly less than the original walking gait. This indicates significant improvement of dynamic stability of the robot. This result also agrees with the objective function of the optimization process in (6). The optimized walking gait and the initial walking gait share some similarities however with the optimized gait, the robot walks with larger step but slower walking speed than the original walking gait. The two walking gait are shown in Fig.

6 9. We found that the optimized walking gait has a different balance strategy with the original walking gait. For the original gait, the robot quickly swings the whole body on the side of the support leg while it moves the swing-leg forward. For the optimized gait, the robot leans on the side of the support leg, blends the torso back in the opposite direction while it moves the swing leg forward slowly. With the optimized gait, the robot also keeps its torso straight up in the vertical direction all the time (a) Original walking gait (b) Optimized walking gait Fig. 9. Testing optimized walking gait on the Fujitsu HOAP robot. V. CONCLUSION We have proposed a methodology to perform optimization of whole body of humanoid robot dynamics based on sensorymotor mapping in a low dimensional space. The key contribution of our work is the sensory-motor mapping in low dimensional space which greatly reduces computational complexity of the optimization process. We obtain non-linear dynamic compensation of the biped locomotion based on a purely learning approach. We obtained our results with both a simulator and a real humanoid robot. Our result also generates a novel set of postures for a humanoid motion that has superior dynamic performance compared to original one. We also observe that learning a new posture by using actions constraints of the action subspace embedding facilitates avoidance of self-intersecting postures. The initial humanoid motion pattern in this paper is assumed to be stable. We are looking forward to extend this framework to allow learning from an initial motion pattern that is not dynamically stable in the near future. Michel for close cooperation and support in using Webots to simulate the HOAP robot [6]. REFERENCES [] M. Vukobratovic and B. Borovac, Zero moment point-thirty-five years of its life, International Journal of Humanoid Robotics, vol., no., pp , 4. [] S. Kajita and K. Tani, Adaptive gait control of a biped robot based on real time sensing of the ground profile, IEEE Conference on Robotics and Automation, pp , 996. [3] J. Yamaguchi, N. Kinoshita, A. Takanishi, and I. Kato, Development of a dynamic biped walking system for humanoid: development of a biped walking robot adapting to the humans living floor, pp. 3 39, 996. [4]R. Sutton and A. Barto, An Introduction to Reinforcement Learning, MIT Press, 998. [5] Y. N. M. Okada, K. Tatani, Polynomial design of the nonlinear dynamics for the brain-like information processing of the whole body motion, in IEEE International Conference on Robotics and Automation,, pp [6] M. Kirby and R. Miranda, Circular nodes in neural networks, Neural Comp., vol. 8, no., pp. 39 4, 996. [Online]. Available: [7] A. J. Ijspeert, J. Nakanishi, and S. Schaal, Trajectory formation for imitation with nonlinear dynamical systems, in IEEE/RSJ International Conference on Intelligent Robots and Systems,, pp [8] K. F. MacDorman, Feature learning, multiresolution analysis, and symbol grounding, vol., 998, pp [9] T. Inamura, I. Toshima, and Y. Nakamura, Acquiring motion elements for bi-directional computation of motion recognition and generation, in Siciliano, B., Dario, P., Eds., Experimental Robotics VIII. Springer, 3, pp [] O. C. Jenkins and M. J. Mataric, Automated derivation of behavior vocabularies for autonomous humanoid motion, in AAMAS 3: Proceedings of the second international joint conference on Autonomous agents and multiagent systems. New York, NY, USA: ACM Press, 3, pp [] R. Chalodhorn, K. MacDorman, and M. Asada, Automatic extraction of abstract actions from humanoid motion data, 4. [] K. F. MacDorman, R. Chalodhorn, and M. Asada, Periodic nonlinear principal component neural networks for humanoid motion segmentation, generalization, and generation. in ICPR (4), 4, pp [3] K. Grochow, S. L. Martin, A. Hertzmann, and Z. Popovic, Style-based inverse kinematics, ACM Trans. Graph., vol. 3, no. 3, pp. 5 53, 4. [4] M. Ogino, Y. Katoh, M. Aono, M. Asada, and K. Hosoda, Reinforcement learning of humanoid rhythmic walking parameters based on visual information, in Advanced Robotics, vol. 8, no. 7, 4, pp [5] K. J. Lang, A. H. Waibel, and G. E. Hinton, A time-delay neural network architecture for isolated word recognition, Neural Networks, vol. 3, no., pp. 3 43, 99. [6] Webots, commercial Mobile Robot Simulation Software. [Online]. Available: ACKNOWLEDGEMENT This research is supported by an NSF Career grant, an NSF AICS grant, an ONR YIP award, and a Packard Fellowship to RPNR. The authors would like to thank Olivier