The 2006 sharpkungfu Team Report

Transcription

1 The 2006 sharpkungfu Team Report Qining Wang, Chunxia Rong, Lianghuan Liu, Hua Li, and Guangming Xie Intelligent Control Laboratory, Center for Systems and Control, College of Engineering, Peking University, , Beijing, China, Abstract This report is a technical documentation of the four-legged robot soccer team sharpkungfu with the specification of our work in This year, our sharpkungfu Team participated in the Technical Challenge of RoboCup In this event, our Medal Awarding challenge got the 8th place in the Open Challenge and 16th in the sum. In October, we got the champion in RoboCup China Open We focus our research on collaborative localization in dynamic environment, quadrupedal gaits optimization and intelligent behaviors. To improve robot self-localization under more natural conditions, we create an experience-based collaborative localization approach to help robot localize under more natural conditions. Moreover, a Particle Swarm Optimization approach to generate high-speed quadrupedal gaits is proposed. In behavior module, our dynamic limit cycle based approach for obstacle avoidance is shown in detail. To perform better under the rule 2006, we create our own set of special actions and correlative strategies. Some modules are still under developing and we are looking forward to participating in the RoboCup 2007 to perform in real world-class games. 1

2 Contents 1 INTRODUCTION 4 2 LOCALIZATION Human Cognition Inspired Localization Landmark & Experience Based Markov Localization Constructing Experience Incorporating Experience In Markov Localization Collaborative Approach The Notion Of Dynamic Reference Object Multi-Robot Markov Localization Experimental Results Localization Experiment Environment Individual Robot Localization Collaborative Localization Discussion GAIT OPTIMIZATION Particle Swarm Optimization Overview of the Basic PSO Improved PSO with Inertia Weight Optimization Problem Control Parameters Initialization Evaluation Modification Experimental Results Experimental Environment Results for Gaits Optimization BEHAVIOR Real-Time Obstacle Avoidance Perform Near Border Dribbling Ball Special Actions Chin Control

3 5 OPEN CHALLENGE Purpose Requirement Process Technical remarks and future work PUBLICATION 30 7 CONCLUSIONS AND FUTURE WORKS 31 8 ACKNOWLEDGMENTS 32 3

4 1 INTRODUCTION This year, our sharpkungfu Team has participated in the Technical Challenge of RoboCup In this event, our Medal Awarding challenge got the 8th place in the Open Challenge and 16th in the sum. In this October, we got the first place in RoboCup China Open We won the WrightEagle Team 2:0, who was the 5th in RoboCup In the last game of the event, we won the TJ-Ark Team 3:0, who entered the second round of RoboCup We focus our research on collaborative localization in dynamic environment, quadrupedal gaits optimization and intelligent behavior. To improve robot self-localization under more natural conditions, we create an experience-based collaborative localization approach to help robot localize under more natural conditions. Moreover, a Particle Swarm Optimization approach to generate high-speed quadrupedal gaits is proposed. In behavior module, our dynamic limit cycle based approach for obstacle avoidance is shown in detail. To perform better under the rule 2006, we create our own set of special actions and correlative strategies. Our code this year is based on GT2004, which is shared by the German Team. As a team, many people contributed to our sharpkungfu 2006, including: Undergraduate students: Kaituo Zhou (major in Mathematics of Information) Yan Huang (major in Mechanics and Engineering) Jiajie Guo (major in Mechanics and Engineering) Fangbai Li (major in Mechanics and Engineering) Master students: Lianghuan Liu (major in Robotics and Automation) Chunxia Rong (major in Robotics and Automation) Hua Li (major in Robotics and Automation) Xin Guan (major in Industrial Engineering and Management) Ph.D. student: Qining Wang (major in Robotics and Automation) Associate Professor: Guangming Xie Professor: Long Wang In the following section, we show the improvement in localization with specification of human cognition inspired collaborative approach. In section 3, we present the work of our team in quadruped gaits optimization, especially of how to implement Particle Swarm Optimization approach in high-speed forward gaits generation. In section 4, strengthened collaboration among teammates is introduced. Moreover, improved obstacle avoidance method based on dynamic limit cycle in sharpkungfu 2006 is presented. Section 5 concerns details of our Open Challenge project in In section 6, we list the selected papers that our team members published this year. At last, we describe the outlook of our ongoing research and expectation to participate in the RoboCup

5 2 LOCALIZATION In human cognition procedures, real-time object recognition is not the only tool to feel the surroundings. People often take tiny features in the view as experience kept in mind. When go to a place that had been to, people may not only depend on actual landmarks, but also exploit features in the environment to localize. The integration of visual experiences and object recognition actually exist in human brain which is mentioned in [5]. Besides, people can get own global position roughly in mind using the information obtained by eyes. Using this known position, locations of surrounding objects can be calculated. If at the same time, another one wants to know own current position, the object can be used as a reference to get the global coordinates. It is natural in human society, especially for team work in complex environments. To apply these mechanisms in real robot systems, it means that a robot can localize among a group of robots in a dynamic environment by experience as well as reference object with no need of knowing exactly who is nearby. To imitate human behaviors or even cognition procedures, most intelligent robots are designed to localize actively and automatically. However, it seems hard for robots to localize in a complex terrain especially an unknown one. To solve this problem, several methods have been applied for localization these years. They mainly differ in using probability to present the possible position and applying various types of sensory information. For example, [4] creates the probabilistic approach based on Markov chain, named Monte Carlo Localization (MCL). For vision-based robot, [3] applies the landmark based MCL to localize in dynamic environment. [8], [9] present the cooperative methods for autonomous position estimate. Whereas those landmark-based methods seem not sufficient if there is no such well recognized landmark considering the odometry error especially when a collision occurs. In that circumstance, the convergence of probability update procedure using MCL is not so satisfied. It may take quite a long time to work out the correct position. A recent work explained in [7] uses the approach of combining an image retrieval system with the Monte Carlo localization. However, the computational cost of this approach is expensive. Besides, the requirement of building a huge database is not so practical, especially in the complex environment. The cooperation in self-localization among multiple robots has many applications in real robot systems (see [6] for overview). For instance, [10] introduces a method for multi-robot localization with certain preconditions. Such robot systems need to identify individual robots. It is quite difficult to perform collaborative localization for robots dealing with situations where they can detect but not identify other robots. In addition, taking the uncertainty of sensors into account, the result of detecting individual robot is not so reliable. Those limitations of the approach make it not so applicable for real robots localizing in complex environments. In the sharpkungfu2006, we create a human cognition inspired collaborative approach that combines image database for experience without landmarks and real-time sensor data for a group of vision-based mobile robots to estimate their positions. We use the team message of dynamic reference object to improve the Markov localization for multiple mobile robots. On the one hand, our approach presents a fast and feasible system for vision-based mobile robots to localize in the dynamic environment even if there is no such recognized landmark to help. On the other hand, we show the collaborative method with introduction of Dynamic Reference Object to improve the accuracy and robustness of self localization, even in the circumstance that the robot can not localize individually 5

6 or has no idea of who is nearby. 2.1 Human Cognition Inspired Localization For the vision-based robot, camera is the main tool to perceive the environment. Image jittering may cause the uncertainty in landmark recognition. The problem of vision system vibration exists in many real robot applications, even the wheeled one in [9] with two-dimensional motion. It has been proven that probabilistic approaches is powerful in robot self localization. However, collision and landmark perception error may cause the result unreliable using such approaches. Actually, the cognition procedure of human beings uses not only those real-time recognized information in images, but also the experience kept in mind. When people get to a place had been to, some small feature will inspire the experience about possible positions. It should be strengthened that the feature acted as a trigger may not necessarily be the actual landmark, such as a beacon. The color distribution and relative position of different pixels in images can also be the clue to percept self position. From this point of view, we integrate an on-line subsystem and an off-line subsystem combined with landmark based Markov localization to help mobile robot to localize in complex situations Landmark & Experience Based Markov Localization Most robot localization systems use landmarks as the tool to predict and correct current positions of mobile robots. Based on Monte Carlo Localization, [3] propose and improve the landmark based Markov localization. In this approach, the current position of the robot is modelled as the density of a set of particles which are seen as the prediction of the location. Initially, at time t, each location l has a belief: Bel t (l) P (L (0) t = l) (1) To update the belief of robot possible location, at first, this approach uses the new odometry reading o t : Bel t (l) P (l o t, l )Bel t (l )dl (2) If robot receives new sensory information s t, then it updates the belief with α being the normalizing constant: Bel t (l) αp (s t l)bel t (l) (3) Considering the mobile robot with three-dimensional complex motion, let the geometric center of robot body as the location vector φ, which contains the x/y/z- global coordinates of the center point. Another vector θ is defined as the heading direction. Then every particle is updated by the motion model as follows when the robot moves: φ t = φ t 1 + t (4) where t represents the displacement in x/y/z coordinates and heading direction. To imitate human cognition features, we divide the sensory update in Markov localization into two parts: updating position probability by landmark perception and experience matching. If the robot recognizes landmarks well enough, landmark based sensor model will update the belief of position with the new landmark reading s t : Bel t (φ t ) βp (s t φ t )Bel t (φ t ) (5) 6

7 where β is a normalizing constant. It is natural that the robot may miss some landmarks with real-time recognition for a period. Thus, we set N 1 (t) which is the amount of lasting frames of having no landmark perception from t as a condition to activate the experience system. If N 1 (t) is great enough, the experience based sensor model will update the probability as follows with e t being the new reading experience with γ being the normalizing constant different from β: Bel t (φ t ) γp (e t φ t )Bel t (φ t ) (6) Constructing Experience The feature that is exploited from images with no landmark in the view, and represents the invariant character of images obtained at positions where collisions and other negative effects more likely occur is defined as Experience. There are various methods for constructing invariant features in images mentioned in [12]. To construct the experience, we use the feature A[f](M) which represents the feature of a given image M calculated as follows: ξ N 2 M[ t 0, t 1 ]M[sin ϕ t 0, cos ϕ t 1 ]dϕdt 1 dt 0 (7) t 0 t 1 ϕ where ξ is the tuning factor. And N is the resolution of the image. In this feature, we choose the pixels neighborhood to be the complex function f[m], in the following form: f[m] = M[0, 0][0, 1] (8) Features are calculated from images collected in certain place where the robot needs experience to help. We construct experience database embedded in robot s memory. This database stores the feature along with the global coordinates of the position where the image is taken. All the features are calculated off-line and stored in the database as experience. When the experience module is activated, the feature of current image taken by camera is computed on-line notated as imagefeature. Meanwhile, the record notated as bestrecord whose feature is most similar to imagefeature is selected from the database. Fig. 1 shows the result of finding the best pose in database based on experience. The query image is on the left while its most similar image in the database is on the right. Their poses (x, y, z, θ) are listed below. When the experience module is activated, difference between imagefeature and the feature of bestrecord is calculated. If the difference is small enough, the pose of bestrecord is transferred into bestpose notated as l best which is in the form of world coordinates in the robot system. With such bestpose, probabilities of all the sample poses are updated and new pose templates which are random poses near the bestpose are generated to perform the resample procedure in Markov localization. It is true that the more experience in database, the more precisely the calculation is. However, building such database is expensive in time cost and even unreachable in complex environments. As a part of the sensor update module, experience can help the Markov localization converge as soon as possible, which means the robot can know own position immediately. In our approach, we only need to construct the database in those really difficult situations. This method works well in real robot applications. 7

8 (a) (2550, 1320, 100, 110 ) (b) (2600, 1400, 100, 90 ) (c) (2100, 500, 100, 135 ) (d) (2300, 600, 100, 120 ) Figure 1: Examples for finding the best pose in image database. Images in the database are collected in a corner without any landmark every 300mm in x, 280mm in y and 30 in θ while z keeps the same. (a) is the current image taken by robot s camera. (b) is the most similar picture to image (a) in experience database decided by the invariant feature mentioned in equation (7). The location error is 50mm in x, 80mm in y, and 20 in θ. (c) is the random sample image taken after (a). The location error in experience image (d) is 200mm in x, 100mm in y, and 15 in θ Incorporating Experience In Markov Localization In Markov localization, every sample pose has a belief which represents the probability of predicted position. In our human cognition inspired approach, sensor module, including landmarks reading and experience, updates the probability using the following equation: p i (t) = K j=1 q (j) i (t), i [1, S] (9) where K is the sum of sensor module types, while S is the number of all sample particles. Every q (j) i (t) describes the position probability at time t using certain type of perception. Specifically, to incorporate the experience module in Markov localization, we set q (k) i (t) as the quality for experience perception to every sample pose. The sum of the dimensionless distance and the dimensionless angle between bestpose and the sample pose is used as a criterion to update the quality with the fact that the quality is higher if the sample pose is nearer to bestpose. The experience quality of every sample pose is in the form of the following equation: q (k) i (t) = q (k) i (t 1) η, v < q (k) i (t 1) η q (k) i (t 1) + ξ, v > q (k) i (t 1) + ξ v, other. where η and ξ are constants used for tuning quality not to change too fast. Thus, the quality can be controlled in a certain range. The criterion v is defined as follows: (10) v = e (σ+τ)2 (11) Here σ is the dimensionless distance between bestpose and current sample pose, while τ is the dimensionless angle. Supposing that current sample pose is l c (x c, y c, z c, θ c ) and the bestpose is l best (x b, y b, z b, θ b ), then σ and 8

9 τ are calculated in equations below: σ = (xc x b ) 2 + (y c y b ) 2 + (z c z b ) 2 D 0 τ = θ c θ b A 0 (12) where D 0 and A 0 are the reference constants which are used to control qualities of σ and τ. Normally, σ=0.05, τ=0.1. Algorithm 1 shows the probability update procedure of experience module in detail with constants D 1, D 2, A 1, A 2 to tuning σ and τ respectively. Algorithm 1 Update probability by experience 1: procedure UPDATE BY EXPERIENCE(pose) 2: for all pose do 3: if σ > D 1 then 4: σ = σ + D 2 5: end if 6: if τ > A 1 then 7: τ = τ + A 2 8: end if 9: setprobability( pose ) 10: end for 11: end procedure Moreover, we set sudden increases of both σ and τ in order to reduce greatly the qualities of the sample poses that are very far away from bestpose. Using such method, the procedure of resample can be more effective and efficient. Those useless particles can be eliminated as soon as possible. The time cost of the Markov localization convergence is relatively satisfied. Fig. 2 describes the relations between q (k) i (t), σ and τ. Incorporating experience in landmark based Markov localization makes the probability update procedure more robust, especially when collisions or other negative effects occur. 2.2 Collaborative Approach The Notion Of Dynamic Reference Object It is natural for human beings to estimate self position by using a reference object. Static reference objects like beacon, and tower can be used to help localize in complex environments. However, global coordinates of such objects need to be known beforehand. Those static reference objects are not applicable in an unknown environment. To solve this problem, we propose the concept of Dynamic Reference Object. The object that can be detected by more than one robots among the team will be the candidate dynamic reference object. If the frequency of clearly recognizing the object is high enough, it may be set as the dynamic reference object. There is no need to know the object s position as a precondition. If a robot can localize itself accurately, the position of the dynamic reference object calculated by this robot is reliable. Meanwhile, another robot that has seen the 9

10 Figure 2: Relations between quality, distance and angle, where D 0 =2000mm, A 0 =1800, D 1 =0.45, D 2 =0.3, A 1 =0.4 and A 2 =0.2. The angle changes from 0 (the top curve) to 180 (the bottom curve) with the increase of 10. Note that the quality drops suddenly when the distance reaches 900mm or the angle reaches 72. Then those unreliable sample poses can be eliminated in resample procedure. reference object can use this calculated position of the object to measure own location. This information is useful for decreasing the time cost of Markov localization convergence and improve the result of position estimate especially for multiple robots collaboration. As far as we know, no explicit research of using dynamic reference object model in Markov localization for multiple mobile robots has been published. There are several challenges to implement this approach in real robot systems. First of all, every robot that has detected the object will broadcast the calculated position to every other robot. Then the robot that needs help may be not able to figure out which position is correct. In addition, the result of the reference object position calculated by a robot may be wrong when another robot needs this information to measure own location. Time delay of the communication is another problem which may bring negative effect to the measurement. To solve problems mentioned above, with the assumption that robots can communicate with each other, our approach integrates Reference Object Position Possibility in the team message which will be broadcasted to every robot. The item which is relevant to the object position in team message includes calculated position, robot ID, time, and position possibility. This position possibility is due to the accuracy of the robot self localization. In our human cognition inspired localization, landmark and experience are key factors. The frequency of detecting landmarks or exploiting experience are used to represent the accuracy of self position estimate. From this of view, the object position possibility notated as P r is measured by the following equation: P r = P l e µ2 + P e e ω2 (13) where P l and P e are certain probabilities for landmark and experience update respectively. µ is the sum of lasting frames after detecting the latest landmark, while ω is the sum of lasting frames after exploiting good experience. In real robot application, P r will be normalized less than 1. If P r is high enough, the calculated result by this robot will be the most reliable one among different robots perception. A robot that needs help always uses the most possible position of the reference object at the same time when it detects the object by itself. To illustrate the method, a common robot system is shown in Fig. 3 with five mobile robots. Object O is supposed to be the 10

11 dynamic reference object. Table 1 is the real-time information in team message of the system in Fig. 3. (a) t=t 1 (b) t=t 2 Figure 3: A simple system with five mobile robots and a dynamic reference object: (a) At time t 1, robot A, B and E can see the dynamic reference object O. They all use their own perception to calculate the position of the object and broadcast to every robot in the team. If at this time robot A, for example, needs the reference object to help, A will use the calculated position of the object from B or E. Querying the most possible position in team message shown in Table 1, A will take the calculated result by B as the reference. (b) At time t 2, C and D have not detected any landmark or experience for a period. Thus their answers to the object position is relatively unreliable. Position possibilities of them are shown to be low in Table 1. The reference object position will be set as B percepts. Table 1: Team message relevant to dynamic reference object Calculated Position Robot ID Time Position Possibility (2388, 700) A t (2264, 658) B t (2530, 710) E t (2368, 803) A t (2401, 801) B t (2103, 743) C t (2215, 725) D t Multi-Robot Markov Localization Incorporating sharing data generated by robots in the same team in Markov localization is the key idea of multirobot localization which may improve the accuracy and efficiency of the probabilistic approach. Common approaches for applying collaboration in multiple robots system normally have the assumption that robot can identify other robots. For example, [10] presents a collaborative method which has limitations that the robot must know who is exactly standing nearby and have a relatively high computational ability. Those limitations may effect applications of real robot systems in negative way. In our approach, the dynamic reference object can 11

12 perform better in collaboration especially in dynamic environments. Robots only need to recognize the object instead of identify all the robots in the team. To illustrate how to integrate the dynamic reference object module in Markov localization, let us assume that robot i uses the reference object position calculated by robot j. Then robot i updates own position belief as follows with a normalizing constant ε: Bel (i) t (φ (i) t ) εbel (i) t where r t is the dynamic reference object position. (φ (i) t )P (φ (i) t approach is similar to the one in the experience model mentioned in equation (10). r t )Bel (j) t (φ (j) t ) (14) The specific probability function using for collaborative In our approach, collaboration is a part of probability update modules in Markov localization. There is a problem that robots should known when to activate the collaboration module using the dynamic object as a reference. To improve Markov localization using our collaborative approach, the collaboration module will be activated in two situations. We set N 2 (t) by using as the sum of lasting frames of having no landmark perception or experience as a condition to activate the collaboration system. If N 2 (t) is great enough and the robot has detected the dynamic reference object, the collaboration module will update the probability of every poses. In addition, if the robot has a perception of the object which has a relatively high position possibility, the robot will use this reference to improve the Markov localization in a collaborative way. 2.3 Experimental Results Localization Experiment Environment The collaborative approach inspired by human cognition presented above has been implemented on the Sony Aibo ERS7 legged robot in RoboCup environment. There are four sequenced color beacons, two goals colored in yellow and sky blue separately, and white lines. Robots must recognize those landmarks to localize real-time and activate different behaviors. When the localization experiment is conducted, the robot performs a scanning motion with its head (pan range [ 45, 45 ]) to search landmarks or exploit experience Individual Robot Localization In this experiment, we use only one four-legged robot to perform localization in RoboCup environment. The robot is placed in the center of the field at first. Then we let robot walk to one corner of the field. On the way to the corner, we add artificial collisions to effect the odometry in a negative way. Specifically, we pick the legged robot up for a while. This procedure makes the odometry not so reliable to imitate real dynamic environment outdoor. After that, the robot stands at one point (shown in Fig. 4) where both landmark perception and experience exploiting can be activated. Then robot performs a head scanning motion to test different localization approach. If we only use landmark based MCL, specifically detecting beacons as new sensory information, the probability distribution converges not satisfied. The result of only using landmark based localization is shown in Fig. 5(b), where the robot can not get the localization result immediately. Then we use our hybrid system with landmark and experience based Markov localization where the experience database is constructed as Fig. 3 shown. Using 12

13 (a) (b) Figure 4: Real position for individual localization experiment. The solid symbol on the field presented in (a) is the real position of the robot. When stands at this position, the robot can obtain vision information using head camera within the head pan range. (b) is selected images in the view at the position when camera heading directions are 45, 10, 10 and 45. our approach, the robot can localize well after 5-10 seconds on average. The experimental result is shown in Fig. 5(c). (a) t = 0s (b) t = 10s (c) t = 10s Figure 5: Comparison of particle distribution between our approach and only landmark(beacon) based method. Solid arrows indicate MCL particles(100). The calculated robot position is indicated by the solid symbol. (a) is the initial uniform distribution. (b) is the calculated result of using only landmark based MCL. (c) is the result of applying our approach Collaborative Localization In this experiment, the orange ball used in the four-legged league is considered as the dynamic reference object. We use three robots to perform multi-robot localization. Every robot uses the hybrid system tested in the individual experiment mentioned above. We set one of the three robots as a sample to estimate our collaborative approach. The other two robots move randomly to catch the ball and broadcast the ball position with position possibilities mentioned in section 2.2. We receive the calculated result from the sample robot. To imitate the outdoor environment, this robot stands in a certain position on the field where we eliminate the landmark which the robot can easily detect. Only experience and collaboration can help the robot localize. The localization result 13

14 (a) t = 0s (b) t = 3s (c) t = 9s Figure 6: The localization result of applying collaborative approach with dynamic reference object. Solid arrows indicate MCL particles(100). The calculated robot position is indicated by the solid symbol. (a) is the initial uniform distribution. (b) is the calculated result after 3 seconds. (c) is the well localization result after 9 seconds. of the sample robot which has used the collaborative approach is shown in Fig. 6. The probability distribution can converges quickly after 3-9 seconds when the dynamic reference object is taken into account Discussion In real robot experiments, we have shown the positive result for legged robot localization using our human cognition inspired collaborative approach in sharpkungfu With limit experience, robot can perform better for localization in RoboCup environment. In collaboration, the ball with unique color was considered as the dynamic reference object. Experiments showed the reliability of our approach in dynamic environment with collisions and sudden position changes. Challenges still exist in real robot applications. The construction of experience database is not so satisfied considering the time limitation and complex environment information. Our future work will focus on how to create experience quickly and easily. Further more, experience and collaboration can be learned by robot itself. We expect to make robot learn more from each localization process. Experiments will be continued in more complex environment with no symmetry. 14

15 3 GAIT OPTIMIZATION As basic behaviors of animals and human beings, walking and running are performed voluntarily and naturally. Over the past years, there are plenty of publications in the biomechanics literature that explain and compare the dynamics of different high-speed gaits including gallop, canter, bound, and fast trot (eg. [30], [31]). To perform legged locomotion, various robot systems have been created (eg. [19], [20], [21]). However, most of the high-speed machines have passive mechanisms which may be not easy to perform many different gaits. Such limitation makes current control method for quadruped robot locomotion stay simple and unstable, in some case, even uncontrollable. Thus, many commercial legged robot systems still use motor as the motion driver. From that point of view, though some early applications have created, how to generate optimized high-speed dynamic walking and running gaits needs further study. To understand and apply the dynamics behind high-speed gaits, researchers have implemented different algorithms or hand-tune methods in simulation [22] and real robot applications [18], [23], [24]. Much published research in learning gaits for different quadruped robot platforms uses genetic algorithm based methods. Different from genetic algorithms, Particle Swarm Optimization (PSO) mentioned in [17], [28], [29] eliminates the crossover and mutation operations. Instead, it uses velocity for each solution to follow the best ones by. PSO can be implemented in a few lines of computer code and requires only primitive mathematical operators. Thus, it is computationally inexpensive in terms of both memory requirements and speed. Taking the memory and processing limitation onboard into account, PSO is more appropriate in gaits learning comparing with the genetic algorithm based methods for quadruped robot, especially those commercial robots with different kinds of motors. Our research focuses on the gait optimization of legged robot with motor-driven joints. The commercial available quadruped robot, namely the Sony Aibo robot, which is the standard hardware platform for RoboCup 4-legged league, is the main platform that we analyze and implement algorithms. Aibo is a quadruped robot with three degrees of freedom in each of its legs. The locomotion is determined by a series of joint positions for the three joints in each of its legs. Early gait learning study used these joint positions to be the parameters to define a gait, which was the case when the first attempt to generate learned gait for Aibo. However, these parameters fail to display the gait clearly and lack of consistence in representing the gaits. Most of recent research used higher lever parameters to represent the gait which focus on the stance of the body and the trajectories of paw. An inverse kinematics algorithm is then used to convert these higher lever parameters into joint angles. The generally high lever parameters used to describe the gait for Aibo can be divided into three groups. One group is for determining the gait patter by changing the relative phase for each leg. [26] mentioned that there exist eight types of gait patters for quadruped animals in nature. [23] described three of the most common gaits for quadruped robot especially for Aibo, which are the crawl, trot and pace. Among them, the trot gait is adopted by almost all team in RoboCup 4-legged league because it is able to generate high speed and at the same time stay stable. In this trot gait, the diagonally opposite legs are synchronized while the two sets of the synchronized legs are 180 out of phase. Another group of the parameters is used to determine the stance of robot. Such as the height of hip or chest above the ground, the sideways distance between shoulder and paw, the distance of the paw from shoulder to front. The last group of parameters describes the locus of the gait. Most of the gaits 15

16 developed for Aibo based on this high lever parameter represent method differ in the shaped of the locus of paws or the representation of the locus, that is the actual parameters used to describe the locus. For example, runswift 4-legged robot soccer team from the University of New South Wales achieved the fastest known hand-tuned gait with trapezoidal loci using the high-level approach described above. They subsequently generated an even faster walk via learning. A team from Germany has created a flexible waking implementation method that can use a variety of different shapes of loci [13]. Later they created a more complex three dimensional gait [25]. In sharpkungfu 2006, we implement the Particle Swarm Optimization based approach in high-speed gaits generation of quadruped robot. In the following paragraphes, an overview of basic PSO and improved PSO with inertia weight is introduced. Our gait learning method is based on Improved PSO with Inertia Weight mentioned in [28], [29]. With the knowledge of using higher lever parameters to represent the gait which focus on the stance of the body and the trajectories of paw, an inverse kinematics model is presented. Moreover, the control parameters and optimization problem are proposed. In addition, how to implement PSO in quadruped gaits learning is introduced in detail. The whole learning process is running automatically by the robot with onboard processor, and no human intervention is involved except changing battery. In real robot experiments, we achieved an effective gait faster than previous hand-tuned gaits, using Aibo as the testing platform. 3.1 Particle Swarm Optimization Overview of the Basic PSO Particle Swarm Optimization (PSO) is a stochastic optimization technique, inspired by social behavior of bird flocking or fish schooling [27]. It is created by Dr. Eberhart and Dr. Kennedy in 1995 [17]. The PSO was derived from study of hunting behavior of bird flock. When a flock of birds are searching for food randomly, each bird decides its flying direction based on two aspects: one is the information of food that itself has acquired, the other is information of directions of other birds. When one of the birds find food, the whole flock can find food. The solution of a problem responds to the position of each bird (called particle) in the searching space. In this paper, the position is represented by a set of parameters for gaits. Bird flocking is assumed to optimize a certain objective function. Each particle has a fitness value which is determined by the objective function. Each particle knows its best value so far with the highest fitness (pbest) and its current position. This information is an analogy of personal experience of a particle. Beside that, every particle can be informed of the best value obtained so far by any particle in the neighbors of that particle, and lbest represent that location. This information is an analogy of an particle knowing how other particles around it have performed. When a particle takes the whole population as its topological neighbors, the best value is a global one and is called gbest. that is the best one among the pbest of all particles. Each particle tries to modify its position using the concept of velocity (determining the direction and distance of the position changing). The velocity of each particle can be updated by the following equation: Where v k i v k+1 i = v k i + c 1 rand 1 (pbest i x k i ) + c 2 rand 2 (gbest x k i ) (15) is velocity of particle at iteration k, c 1, c 2 are acceleration factors. rand 1 and rand 2 are random numbers between 0 and 1, x k i is current position of particle i at iteration k, pbest i is the pbest of particle i and 16

17 gbest is the best value so far in the group among the pbest of all particles. The current position (search point in the solution space) can be modified by the following equation: x k+1 i = x k i + v k+1 i (16) Improved PSO with Inertia Weight Using inertia weight can control the influence of previous velocity on the current velocity. The update equation for velocity with inertial weight is as follows: v k+1 i = w v k i + c 1 rand 1 (pbest i x k i ) + c 2 rand 2 (gbest x k i ) (17) Where w is the inertia weight. PSO with larger inertial weight results in better global searching ability while little inertial weight improves local searching ability. Empirical result shows that PSO has faster convergent speed when w is in the range from 0.8 to 1.2. In order to search fast at first and search carefully at last, w can vary from big to small, similar to the annealing temperature of Simulated Annealing Algorithm. 3.2 Optimization Problem In this section, we describe the optimization problem in high-speed gait learning, with specification of control parameters Control Parameters Similar to GT2004 [13], we use high-level parameters to represent the walking gaits of Aibo. Since our code in 2006 is based on GT2004, our control parameters in gaits optimization are nearly the same with the German Team. Specifically, they are foreheight, forewidth, forecenterx, which describe the center foot position of the forelegs relative to the body of the robot. In addition, hindheight, hindwidth, hindcenterx are parameters for the hind legs which describe center foot positions. Moreover, steplen is the time for one complete step cycle. And groundphase, liftphase, loweringphase define the timing of the step cycle. groundphase defines how much time of the step cycle the foot will be on the ground. liftphase defines how fast the foot will be lifted, while loweringphase defines how fast it will be lowered. There are four more values determining the high and width of the rectangles(the shape of paw locus we choose) for fore and hind legs. We have done some work on the robot s gait, and found out trot gait is almost always the most effective pattern, so we choose the gait pattern to be only trot gait, thus eliminate the dimension of search space and reduce the searching time. For stance parameters, based on our observation and analyze of the motion for Aibo, we conclude that forward-leaning posture can make walking fast, thus we constrain the changing spectrum of stance parameters to keep robot in forward-leaning posture, that is the height of hip higher than that of chest. As to locus, we choose rectangle to be the shape of paw trajectory because it is proved to be effective to generate high speed gait by other teams, but we will try to use free shape lately to search for higher-speed gait. Our task amounts to a parameter optimization problem in a continuous dimensional space. We adopt forward speed as the sole objective function in this paper. We formulate the problem as a Particle Swarm Optimization 17

18 problem by defining each possible set of parameters as a particle, a predetermine number of sets of parameters construct a particle swarm which will expose to learning by implemented PSO, with the forward speed of each parameter being fitness Initialization The first step is generation of the particles in the solution space, which is the possible set of gait parameters. These particles construct a swarm that can be represented by {p 1, Λ, p N } (where N = 10 in this case).these sets of parameters are acquired by random mutation around a previous tune set of parameter. Each parameter set has an initial velocity which is generated randomly within a given range. The length of the range is chose to be the half of the range of the corresponding parameters, which is calculated through limitation of Aibo s structure. Velocity calculated later is also constrained inside the spectrum. If the velocity is too small, the evolution will conform to local maximum. We have tried to use a fine-turn parameter to be the starting particle, and the limited range of the velocity to be fairly small, we failed to find a better parameter set. But if the velocity is too big, the particle may fly beyond global maximum. We set the spectrum to be ±V imax, where V imax = (x imax x imin ), with (x i min, x i max)as the changing range of particle P i. The range we choose turn out to be appropriate to avoid these two problems. To expedite the search for an optimal solution, c 1, c 2 are set to 2, The initial pbests are equal to the current particle location. There s no need to keep track of gbest while it can be acquired from pbests, that is the pbest with the best fitness is gbest Evaluation The evaluation of parameters is performed using sole speed. Since the relation between gait parameters and speed is impossible to acquired, we do not know the true objective function. There is no accurate simulator for Aibo, we have to perform the learning procedure on real robots. In order to automatically acquiring speed for each parameter set, the robot has to be able to localize itself. We use black and while bar for Aibo to localize, because given the low resolution of Aibo s camera, detecting black-while edge is better than other things. During evaluation procedure, robots walk to a fixed position, then load the parameter set need to be evaluated, walk for a fixed time, 4s, stop and determine the current position. Noticing that both before and after the walk with the evaluating parameter set, robot is in static posture, so the localization is better compare to localizing when running. Now the starting and ending location have been acquired from detecting the bar, speed can be calculated Modification After evaluating the speed of each particle P i, the current pbests are compared with the respective particles. If performance of p i are better than pbest i, pbest i is replaced by the new position of p i. After pbests are updated, the new gbest, the best among pbests can be known. With the basic PSO, it s inclined to step into local maximum, because it always follows the best value found so found. It s obvious when the starting particle is near or in the local maximum position. In order to escape from this inclination, we incorporate the concept of simulated annealing by using changing inertial weight. We use the following equation to calculate: 18

19 w = wmax ((wmax wmin /itermax )) inter (18) Where wmax is the initial weight, wmin is the final weight, itermax is the maximum iteration number, and iter is the current iteration number. Also, we adopt HPSO described before to perform the evolution procedure to broaden the search area. The natural selection mechanism is executed after the step updating gbest of all particles. In finding the best solution, the selection is achieved by forming subgroups from the entire group in the solution space. Then, each particle is evaluated, and the best particle is found in each subgroup. Finally, the particles with low performance are replaced by the particle with the best performance in the subgroup. Thus, the new search with the selection mechanism would provide more chance to find an optimal solution. The remained particles, however, are evaluated with the original pbests and gbest. Therefore, the HPSO method realizes more intensive search nearby the best particles [32]. The modification of remain particles is performed by (16),(17),(20) in each iteration. The first term in the right-hand side of (17) is for diversification in the search procedure, which keeps trying to explore new areas. The second and third terms are for intensification in the search procedure. They help in moving toward the pbests and/or gbest [33]. The method has a well-balanced mechanism to use diversification and intensification efficiently in the search procedure Experimental Results Experimental Environment Forward speed that is the optimization object is calculated in Four-legged league environment. Marks for body adjusting in walking are placed at one end of the field. Such marks help Aibo walk in a straight line (see [13] for details). In optimization process, Aibo walks for 4s and determine the covered distance, thus calculating the speed of the parameter set. Figure 7: Marks for body adjusting in forward gaits evolution. (similar to the environment introduced in [13]) 19

20 3.3.2 Results for Gaits Optimization Under the preset environment, Aibo walks in certain distance to optimize the forward gait. Experiment shows that the optimal forward speed converges quickly when using PSO. In our PSO based method, different to many existing gait optimization methods based on Genetic Algorithm (eg. [13], [25]), the initial values of parameters can be selected randomly from a rational range. Those initial values need not any hand-tune parameters. To avoid the motor abrasion of irrational gaits in first two generations, we artificially set fitness (forward speed) to be zero. In our experiment, at the beginning, w is 1 from generation 1 to generation 10. In this period, the global search is Figure 8: Forward gait optimization result. x-axis represents the iterative times, while y-axis is the best forward speed in each generation. The fastest forward speed in this experiment is 416mm/s. strengthened which can help the optimization process converges relatively quickly. At the 10th iteration, we have already achieved 390mm/s gait for Aibo, which is very impressive. From generation 10 to 15, w decrease to zero. In and after this period, our method increases local search ability and optimization process searches limited area around best parameters found before. The optimization result improves slowly but smoothly. Fig. 8 shows the specific result of optimization. 20

21 4 BEHAVIOR Intelligent behavior is one of the main features for autonomous robot. In sharpkungfu 2006, we improve the behavior control module in three parts. First of all, we introduce a new approach for real-time obstacle avoidance based on dynamic limit cycle. Then the new behaviors and strategies for Aibo to perform near border are described in detail. Dribbleing ball is another improvement in our code It can help robot handle ball more rational and reliable taking use of the chin. 4.1 Real-Time Obstacle Avoidance As the essential part of autonomous robot navigation, obstacle avoidance faces increasingly difficulties especially in dynamic and unknown environment. It is quite important to response immediately with real-time sensory information. Common approaches in global navigation, including artificial potential field approach, electrostatic potential field, and the magnetic field method, etc[14], have advantages in self navigation. However, the computational cost in those methods above is relatively expensive. The limit cycle approach has been used in the global-vision based robots on land[15], and under water[16]. It is relatively cheap in computational cost and easy to be convergent. However, most applications need global control and have no consideration of different characteristics of obstacles. Besides, they keep the size of limit cycle unchange all the time. To apply in real autonomous robot, features of different obstacles and dynamic size of limit cycle should be taken into consideration. In sharpkungfu 2006, we introduce time-variable limit cycle to help robot avoid obstacles. To show the approach, we simply describe the shape of Aibo as a cycle in the two dimensional plane. Considering the following nonlinear system for dynamic limit cycle applying in Aibo: x = ρ(ỹ + γ x( 1 4 v2 x 2 ỹ 2 )) ỹ = ρ( x + γỹ( 1 4 v2 x 2 ỹ 2 )) (19) where ρ is the character factor of the obstacle which is set to be a positive value. γ is the convergence factor. And v is the relative velocity to the obstacle which is dynamic when the robot moves. The size of limit cycle is changing when system(21) switches. To prove the circle x 2 +ỹ 2 = 1 4 v2 is the dynamic limit cycle of the switched system(21), we use the common Lyapunov function: V ( x, ỹ) = x 2 + ỹ 2 (20) such that: V ( x, ỹ) = 2ργ( 1 4 v2 x 2 ỹ 2 )( x 2 + ỹ 2 ) (21) For limit cycle, we can see that V ( x, ỹ) < 0 when V ( x, ỹ) > 1 4 v2, while V ( x, ỹ) > 0 when V ( x, ỹ) < 1 4 v2. This shows the following region is absorbing. B = {ρ 1 V ( x, ỹ) ρ 2, 0 < ρ 1 < 1 4 v2, ρ 2 > 1 4 v2 } (22) 21

22 Since this argument above is valid for any 0 < ρ 1 < 1 4 v2, and ρ 2 > 1 4 v2, when ρ 1, ρ 2 get close to 1 4 v2, region B shrinks to the circle V ( x, ỹ) = 1 4 v2.this shows that the circle is a periodic orbit as shown in Fig. 9(a) when v = 280, ρ = 0.01, γ = This periodic orbit is called a limit cycle. We can see the trajectory from any point ( x, ỹ) moves toward and converges to the limit cycle clockwise when close. The counterclockwise (a) clockwise (b) counterclockwise Figure 9: Phase portrait of limit cycle condition can be derived by the following system (shown in Fig. 9(b)): x = ρ( ỹ + γ x( 1 4 v2 x 2 ỹ 2 )) ỹ = ρ( x + γỹ( 1 4 v2 x 2 ỹ 2 )) (23) Considering that the trajectory from any point ( x, ỹ) inside the limit cycle moves outward the cycle, and the trajectory from any point ( x, ỹ) outside the limit cycle approaches the cycle with distance determined by the relative speed v, the limit cycle provides a method for obstacle avoidance among multiple mobile robots. In RoboCup Four-legged League, there are many obstacles during the game. Robots can be considered as motive obstacles. When the dog approaches a teammate holding ball, it must stay out of the area where teammate handles ball, and be ready to perform cooperative strategies. If the dog holding ball encounters an opponent, it must control the ball and quickly avoid the approaching robot, especially when perform kicking ball in front of opponent goalie. Own penalty area is another one that can be taken for an obstacle. If the robot moves parallel to own ground line, it must avoid from walking into the own penalty area. When the dog is in a safe region, by the dynamic limit cycle approach, it will move away the obstacle toward the safe circle with a radius relevant to the speed of the obstacle. Let α denote the orientation of the obstacle, (x 0, y 0 ) the center point of the obstacle. With the following transformation, we get the expression of system (21) in the original frame: x = cos α( x + x 0 ) sin α(ỹ + y 0 ) y = sin α( x + x 0 ) + cos α(ỹ + y 0 ) (24) Let v denote the translational velocity of the dog in the original frame, θ the direction of the motion. The kinematic 22

23 Limit Cycle for Obstacle teammatecycle opponentapproachcycle opponentnotapproachcycle opponentgoaliecycle ownpenaltycycle passingballcycle unknown Obstacle Description teammate handles ball in front opponent approaches to block opponent has no aggressive behavior opponent goalie block ball own penalty area ball passed by other two teammates obstacle used default parameters Table 2: Obstacle percepts list of sharpkungfu 2006 model of the dog is described by: Then we can see: ẋ = v cos θ ẏ = v sin θ ẋ ẏ = tan θ (25) v = ẋ 2 + ẏ 2 θ = arctan ẋ ẏ + α (26) Different obstacles have their own characters, with ρ matching to characters respectively. Using ρ in different values can control the magnitude of the absolute speed. With the dynamic radius of the limit cycle, robot can perform more flexibly and rationally. In sharpkungfu 2006, we consider the following objects as obstacles with different character factor ρ. Table 2 shows the obstacle in detail: Challenges for using the dynamic limit cycle approach mentioned above are: 1) Obstacle Recognition. Precise avoidance depends on the accurate perception of the obstacle. In Four- Legged League, environment is increasingly complex and unfeatured. It is not so easy to figure out what the obstacle is. Using new robot percept mentioned in vision module, we can roughly calculate the position of the robot and the distance. It is reliable to avoid the approaching robot by our dynamic limit cycle approach. 2) Multi-Obstacle Avoidance. As a dynamic obstacle, collision avoidance between multiple robots is challenging. There are difficulties to apply our approach in this situation. Because vision sensing ability of Aibo is quite limited, to figure out different robots when more than one appear in the same image is not so satisfied. We still try to apply new method to solve this problem. 3) Relative Velocity. In the limit cycle system shown in equation(21), relative velocity to the obstacle is quite important for applying suitable radius of limit circle. We use the change of distance to the point on obstacle 23

24 body which is the closet one to the dog as the reference. Then calculate the relative velocity using: distance t distance t 1 t (27) With this relative velocity, robot can perform well when encounter single obstacle. 4.2 Perform Near Border In 2006, new behaviors and strategies correlated to new border line are important. Any unappropriate behavior near border may cause a negative impact. For example, if the ball is near border in own half field, it is dangerous for the defender to handle ball inappositely to let it out of field. Because it may benefit the opponent striker to control the ball. To avoid this situation, we implement the near border behavior. We define that for a player, if the distance to border line is less than 600mm, it enters the near border area. It is simple that if the player handles ball near border, it can hold ball and move it along the direction vertical to borderline. However, actual test shows that different gaits along with grabbing ball motion may not help control ball well. Therefore, we divide the circle area around player into four parts. Fig. 10 shows the different parts of the near border area which may activate strategies respectively. We define the variable robotpose.anlgleto-border which represents the absolute value of angle between robot s body direction and normal line to the border. Area 1 is the place where the angle-to-border is in range from 120 to 180. In area 2 and 3, the angle is between 80 and 120. Area 4 means the angle is less than 80. In area 1, the robot grabs ball and adjusts its Figure 10: Strategy field in near border area body direction first. Then the robot performs a sideways walk moving ball into field. In area 2 and 3, the robot performs a sideways walk directly. Player walks forward directly to the field if enters the area 4. In GT2004, option handle-ball is used for grabbing ball and taking relevant actions. To apply our strategy in near border area, we rewrite the option handle-ball to control ball more appropriately. Fig. 11 shows the states switching in new option handle-ball. If the robot needs to handle ball in near border area, option handle-ball will activate the newly created option handle-ball-near-border, shown in Fig. 12, which specifically deals with the ball near border. In option handle-ball-near-border, robot goes to the ball first and then another new option back-and-turn-and-push is activated. This option executes the basic actions to handle ball in motion level. Fig. 13 describes the details for option back-and-turn-and-push. Applying new behavior and strategies mentioned 24

25 Figure 11: Option handle-ball Figure 12: Option handle-ball-near-border above, robot can play well on the field. And the probability of out-field ball caused by own players is relatively low. 4.3 Dribbling Ball In a soccer game, the team that controls the ball more will probably score more. There are two ways to control the ball: a player s dribbling and two or more players passing. Passing ball between robot players is still a difficult task, as shown in the result of the Passing Challenge Our team is also developing passing ball, but the result is not as good as that in developing dribbling ball. The dribbling ball behavior in sharpkungfu 2006 is based on the gait called walking with ball. We use PSO method (See Section 3 GAIT OPTIMIZATION) to optimize the gait so that the robot can walk with ball smoothly and steadily in any direction: forward, backward, sideward and turning. Moreover, the robot can still see as far as possible when walking with ball, because its uses the chin to control the ball and keeps the head up (See Section 4.5 Chin Control). In the dribbling ball behavior, the robot controls the ball and walks towards the opponent goal. It will avoid obstacles and adjust direction when walking 25

26 Figure 13: Option back-and-turn-and-push with ball. When the time is over (In Rule 2006 robots are allowed to hold the ball for up to 3 seconds), it will kick the ball to opponent goal or pass the ball to its teammate. 4.4 Special Actions GT2004 presents a series of good special actions[13]. Most of them are useful, but they are not exactly applicable with consideration of new rule. New rule uses only white line instead of the border wall. This change not only affects vision and behaviors module, but also asks for appropriate special actions to perform near border, like handling ball. To perform well in new field, we design our own set of special actions instead of those used in GT2004 except the headleft and headright. Table 3 shows the list. Those actions are tested in real games. All of these actions have their own features and advantages. Specifically for example, in action chestsoft robot first holds the ball and then push the ball out. The advantage of this special action is that the ball can be held firmly and the opponent can not easily capture the ball. And we also changed the condition to activate special actions. For instance, when the ball is near the border line, any inapposite action may push the ball out of the field. When the ball near the border, we have two strategies using relative actions. In most cases, we use the method Perform Near Border(already explained in section 6.2). In a few occasions, when the behavior near border can not be activated, the special actions using paw will be inspired, like slapleft or slapright. The activated condition is carefully tested. 26

27 Special Action Name goalieblock chestsoft forwardheadkick-near forwardheadkick-far slapleft slapright slapright-left slapleft-right headleft headright Description open both arms use head to hold ball then push it out head knock to push ball for short distance head knock to push ball for long distance kick ball forward(45 left) using paw kick ball forward(45 right) using paw move ball to the left using paw move ball to the right using paw kick ball left using head kick ball right using head Table 3: Special actions list of sharpkungfu Chin Control Chin Control is another part that we implement in the motion module of sharpkungfu It is designed to perform in Dribbling Ball behavior. In this motion, the robot opens its mouth to an appropriate angle, and holds ball combining with the neck-tilt, the head-pan and the head-tilt motion. Using chin control, camera can be adjust to relatively high position. It is quite useful to avoid losing vision information when robot holds the ball. Fig. 14 compares the actions for grab ball used in different teams. (a) Chin control in sharpkungfu 2006 (b) Grab ball used in GT2004 Figure 14: Actions for grabbing ball in different teams There is a problem in using chin to grab ball. The control area using head to grab ball like b) in Fig. 14, is relatively larger than that one using chin. If the next action is turning body around or moving ball to left or right, the possibility of losing ball control is increasingly high. To solve this problem, we apply head control motion used in GT2004 to perform in the circumstance mentioned above. Chin control is only used in grabbing ball when perform Dribbling Ball behavior. Robot can use chin to help control ball to move forward or backward. 27