A Machine Learning Approach to Developing Rigid-body Dynamics Simulators for Quadruped Trot Gaits

Transcription

1 A Machne Learnng Approach to Developng Rgd-body Dynamcs Smulators for Quadruped Trot Gats Jn-Wook Lee Abstract I present a machne learnng based rgd-body dynamcs smulator for trot gats of the quadruped LttleDog. My contrbuton can be dvded nto two parts: the rst part for the reducton of one-tme-step predcton error, and the second part for a more accurate predcton over longer tme scales. Frst, n order to reduce the one-step predcton error, I compared three regresson methods: st order lnear regresson (LR), lnear weght projecton regresson (LWPR) [] and Gaussan process regresson (GPR) []. Although GPR shows the hghest accuracy, ts cost for computaton and storage O(n ) and O(n ), respectvely s too hgh to handle a large amount of tranng data whch are requred n my problem. Therefore, I developed a sparse GPR, called local GPR (LGPR). In LGPR, tranng data are dvded nto k clusters by k-means, and a locally full GPR model s constructed for each cluster. The predcton s done wth the local model closest to a test data pont, n terms of the normalzed Eucldean dstance. The cost for computaton s tractable, approxmately O(m ) where m=n/k, makng t possble to use a large amount of tranng data. I showed that LGPR sgncantly reduces the one-step predcton error. Second, to reduce the accumulaton of predcton error, I proposed a projecton method, called α- P ROJ. I found that usng LGPR as t s accumulates the predcton error too much over longer tme scales. One of the man sources of such error turned out to be n some predcted states that are not wthn the dynamcally feasble regon. α- P ROJ projects a predcted state to such regon. I dened the regon wthout usng any specc nformaton of LttleDog (e.g. max/mn jont angles). It s only dened by the tranng data. Thus, ths projecton method can be appled to other robot systems wthout modcaton. α-p ROJ hghly mproved the predcton over longer tme scales. The emprcal results show that these key deas led to the sgncant reducton of predcton error and better performance than the smulator. I. INTRODUCTION A. SIMULATOR FOR QUADRUPED TROT GAITS In recent years, Machne learnng algorthms have been extensvely appled to the development of a robotc controller to learn the robot's dynamcs. When developng such controllers, t s very useful to have an accurate Fg.. The LttleDog robot smulator n order to test the controller's performance before applyng t to a real robot. Especally for the ground robots, there have been several rgd-body dynamcs engnes, the most wdely used one beng Open Dynamcs Engne (). Although s used n wde-rangng applcatons not only for robotcs but also for games and anmatons, t s not accurate enough for engneerng purposes. Furthermore, n general, t s very hard to get an accurate dynamcs model Desgned and bult by Boston Dynamcs Open Dynamcs Engne, for complex robot systems []. Varous characterstcs of a robot such as a complex body shape, jont motors and other speccatons, whch can sgncantly affect ts dynamcs, are far too complcated to be nput to an type smulator. In order to resolve these challenges, I propose a learnng based approach to developng an accurate smulator (hereafter, an ML smulator) that does not requre any specc nformaton, but learns from data for a robot of nterest. I focus on learnng dynamcs of trot gats of the quadruped LttleDog on a at terran. B. APPROACH IN THIS RESEARCH Denote by s t a state of the robot at a certan dscrete tmestep t, then our goal s to predct s t+ for t =,,..., T lke the followng: s t+ = predct(s t, u t ). A state s s R N and an acton u s R M, where N = and M = for the problem I deal wth. The ntal state s and actons u t for t T are gven for each tme step. s conssts of the body's poston, lnear velocty, orentaton, angle of jonts and tme-dervatves for them, such that t can solely represent the state of the robot at tme t. Unlke ordnary regresson problems, developng ths knd of ML smulator s challengng for two reasons. Frst, the ML smulator should predct mult-dmensonal outputs for every element of s t+ accurately. Second, a predcted state s t+ must be used as a feature set to predct ts successve state s t+. Ths means that once s t devates from the dynamcally feasble regon, the predcton error wll quckly accumulate or dverge. In order to resolve these dfcultes, I constructed largely two goals. Frst, I focused on reducng the onestep predcton error. I compared varous regresson methods: st order lnear regresson (LR), lnear weghted projecton regresson (LWPR) and Gaussan process regresson (GPR). GPR was the best among them. However, GPR suffers from extremely hgh cost for computaton O(n ) and storage O(n ) such that only thousands of tranng data ponts are actually used for a predcton. Thus, I adopted a sparse GPR, called subset of regressors (SoR) [], [5], [6]. For ths knd of sparse GPRs, t s very mportant to choose a good set of m support data ponts whch can represent the entre n ( m) tranng data ponts. I proposed an efcent method that nds such support data ponts by a k-fold-cv test n the tranng data pool. Ths selecton method outperformed over the ordnary random samplng. However, the accuracy was not good enough to be used as a smulator agan. Thus, I developed a local GPR (LGPR) that dvdes tranng data ponts nto k clusters by k-means, and predct wth only one local model closest to

2 a test data pont. Ths further reduced the one-step predcton error, and the computatonal cost also reduced to a tractable level, O(m ) where m=n/k, whle explotng all data ponts n the tranng data pool. The second part of my contrbuton s on the reducton of the predcton error over longer tme scales (hereafter, the sequental predcton error). The sequental predcton error accumulates as tme step ncreases because each one-step predcton error s not zero. Ths accumulaton can cause a predcted state s t t to devate from the feasble data regon (FDR), n whch robot's dynamc constrants are satsed. I proposed a projecton method called α-p ROJ that prevents ths. I show that the sequental predcton error sgncantly decreased. Secton II descrbes data collecton and feature selecton. In Secton III, varous regresson methods are compared. Secton IV proposes the method of choosng support data ponts. Secton V explans the proposed novel LGPR. Secton VI presents the projecton method α-p ROJ that sgncantly reduces the sequental predcton error. Secton VII presents emprcal results, and Secton VIII contans concludng remarks. II. DATA AND FEATURE SELECTION I collected data from LttleDog's trot gats. Each data pont s encoded as {(s t, u t ), s t+ }, the feature part and the response part. Each element of the feature part (s t, u t ) has ts own scale, so each element s normalzed to have a unt-varance. I splt the collected data nto two categores, T ranp ool and T estp ool, such that all the tranng data come from T ranp ool and all the test data from T estp ool. The tme step was. second and one trot move lasted for seconds so that each move has about data ponts. There were types of moves: F /F L/F R(turnng or movng straght forward), B/BL/BR (turnng or movng straght backward), L/R (movng sdewse), DL/DR (movng dagonally forward) and S (standng). For each type of trot gats, there were moves, for T ranp ool and 6 for T estp ool. Now, pror to the dscusson of the selecton of a proper feature set, let me dene a robot's state s here n more detal. There are rgd-bodes ncludng the robot body, eght leg parts and the ground. The jont constrants among these rgd-bodes restrct the degrees of freedom (DOF) of the system to 8; so, I have 8 dmensonal conguraton space: q = (x, y, z, φ, θ, ψ, ϕ,..., ϕ ). I denote by (x, y, z) the poston of the center of gravty (COG) for the robot body. (φ, θ, ψ) means the roll-ptch-yaw orentaton of the body. ϕ j represents the j-th jont angle for the -th leg, where each leg has three jont angles. Addtonally, I consder the rst order tme-dervatves,.e. the lnear velocty and the angular rate of the orentaton/jont angles. One mght thnk that a state should be wrtten as the followng: s = (x w, y w, z w, ẋ w, ẏ w, ż w, φ, θ, ψ, φ, θ, ψ, ϕ,..., ϕ,...ρ,..., ρ ) R 6, () where the superscrpt w stands for the world frame, b means the body frame, and ρ j denotes the angular rate of each jont. However, takng nto account the fact that features should be relevant to predctng a response, three features (x w, y w and ψ) should be removed from the feature set. A Robot's x- y poston (x w,y w ) does not actually affect robot's behavor, t t+ acton state (gven) ~ z x y z Φ θ Φ θ ψ φ φ j ρ j j z Fg.. x y z Φ θ Φ θ ψ φ j One-step predcton for a mult-dmensonal response and so doesn't the yaw angle ψ. Excludng these features s very mportant to dene the feasble data regon (FDR) because FDR must be bounded. I wll dscuss FDR later n Secton VI. For the smlar reason, ẋ w,ẏ w and ż w should be rewrtten to be orgnated from the body frame. Because the tme step s xed to be.s, I used the dfference terms x b, y b, z b, φ, θ and ψ rather than the velocty terms ẋ b, ẏ b, ż b, φ, θ and ψ. Fnally, a state s can be rewrtten as: s = (z w, x b, y b, z b, φ, θ, φ, θ, ψ, ϕ,..., ϕ,...ρ,..., ρ ) R. () An acton u s smply u = ( ϕ,..., ϕ ) R where ϕ j denotes the desred jont angle for the next tme step. Therefore, there are 5(=+) features and responses. III. COMPARISON OF REGRESSION METHODS For the one-step predcton, the problem I consder has a mult-dmensonal response s t+. As llustrated n Fg, I used a state s t and an acton u t to predct each element of the response s t+. I appled the same regresson method to predct each s t+ (s denotes the -th element of a state s). To reduce the one-step predcton error rst, I compared three regresson methods: st order lnear regresson (LR) as a baselne approach, lnear weghted projecton regresson (LWPR) [] and Gaussan process regresson (GPR) []. LR predcts a mult-dmensonal response Ȳ = X (X X) X Y, where X and Y are a desgn matrx and a response matrx, respectvely. X s a test feature set. LWPR manages K locally lnear models, called a receptve eld, wth a center c k. The predcton can be done by computng the weghted mean of each model: ŷ = K k= w kŷ/ K k= w k. Each model's weght s w k = exp( (x c k ) D k (x c k )/) where D k s a dstance metrc n a Gaussan kernel. LWPR learns tranng data one by one to update ts center c k and parameters, and ncrease or decrease the number of receptve elds f necessary. LWPR s parametrc; so t can predct quckly and handle a large number of tranng data. However, the major drawback of LWPR s the requred manual tunng of many hghly datadependent parameters []. GPR assumes y = f(x) + ε wth addtve Gaussan nose ε wth varance σn. It predcts a response f(x ) = k (K + σni) y wth varance V [f ] = k(x, x ) k (K + σni) k, where k = φ(x ) Σ p Φ, k(x p, x q ) = φ(x p ) Σ p φ(x q ) and K = ΦΣ p Φ. GPR requres less dataspecc tunng of hyperparameters of ts covarance functon [], and can be easly optmzed by common gradent ascent algorthms. However, t suffers from extremely hgh computatonal cost O(n ) and memory usage O(n ) for nvertng (K + σni). Varous sparse approxmatons were appled to full GPR for resolvng ths drawback [7]. I wll ρ j ~ φ j

3 dscuss ths n the followng secton. For GPR, I chose the automatc relevance determnaton (ARD: a squared exponental kernel wth ndvdual dstance metrc for each feature). Then I optmzed ts hyperparameters by usng a conjugate gradent descent that mnmzes the negatve margnal loglkelhood. These optmzed hyperparameters were also used to tune the dstance metrc of each feature for LWPR. I evaluated these three regressers by a hold-out crossvaldaton test wth the dentcal test data set. I used normalzed mean squared error (nmse) as an error measurement metrc such that I can easly compare the error of each element of a state. In terms of test error, GPR outperformed over other two regressors as shown n Table I. Therefore, I decded to use GPR for mplementng the ML smulator. IV. SAMPLE SELECTION FOR SPARSE GPR Due to GPR's hgh cost of computaton and memory, I consdered sparse GPR methods that are desgned for handlng a larger set of tranng data. I tred one of them, called subset of regressors (SoR) [], [5], [6]. The key dea for SoR's approxmaton s that t chooses m `support data ponts' that can represent the entre tranng data pool T ranp ool. In SoR, the rest (n-m) tranng data ponts are assumed to be ndependent wth one another and they only communcate through m support data ponts. SoR's cost for computaton and memory s O(m n) and O(mn), respectvely. Needless to say, t s very mportant for SoR to choose m good support data ponts out of the entre n tranng data ponts. These data ponts should be well-dstrbuted so that t can cover up most of T ranp ool. There are several approaches that jontly nd optmal hyperparameters and support data ponts where m s gven [8]. However, especally for the hgh dmensonal and data-rch problem I consder, I couldn't obtan the soluton n a reasonable tme. Instead, I nvented a smple-but-efcent method that nds support data ponts by dong k-fold-cv test n the tranng data pool T ranp ool. Because I used a squared exponental kernel, covarances among data ponts are closely related to ther normalzed Eucldean dstances. Therefore, f a data pont s not predcted well n a leave-one-out-cross-valdaton (LOOCV) test, the pont s lkely to be separated from other data ponts. To cover up a wder range of T ranp ool, these separated data ponts should be ncluded n the support data ponts set as many as possble. The key dea of my samplng method s measurng `separateness' of each tranng data pont, and gvng them `separateness dstrbuton'. Samplng support data ponts from ths dstrbuton prevents choosng too many from clustered data ponts, such that selected ones can be well-dstrbuted n T ranp ool. LOOCV test s the deal one for my samplng method. However, the problem s that full GPR can hold only thousands of tranng data ponts for each tme of CV test. Thus, I appled k-fold-cv test such that a small number of tranng data ponts are nvolved n each CV test. Frst, I randomly dvded the entre n data ponts n T ranp ool nto k groups, and dd k tmes of CV test by full GPR. n/k tranng data ponts and n(k-)/k test data ponts are nvolved n each tme of the CV test. Each data pont n T ranp ool was tested k- tmes, and hence had k- number of predcton errors. Summng up these error for each data pont and normalzng t over n tranng data ponts gave the `separateness dstrbuton' over the entre tranng data pool. Because I appled ndvdual regresson model for each element of a state s, `separateness dstrbuton' should be computed for each element separately. V. LOCAL GPR SoR gave less predcton error. However, the one-step predcton should be as accurate as possble because the sequental predcton error can be greatly accumulated through a number of one-step predctons. Thus, I nvented a novel sparse GPR called local GPR (LGPR). LGPR can take advantage of usng the entre tranng data ponts. It dvdes T ranp ool nto k clusters by k-means, and construct k locally full GPR models. A predcton for a test data pont s done wth the closest local model only. Ths s smlar to the block-dagonal approxmaton of a covarance matrx. However, t s dfferent n that only one local model s actvated for a predcton, so computatonally less expensve. Of course, the hyperparameters are re-optmzed for each local model. Each local model has approxmately m = n/k tranng data ponts, so the computatonal cost of LGPR wll be O(m ); the overhead to nd the closest local model s gnored. LGPR can be a better alternatve for hgh dmensonal problems whch requre a very large set of tranng data for an accurate predcton. VI. PROJECTION TO TRAINING DATA POOL In the prevous sectons I focused on the reducton of the one-step predcton error. One mght thnk that the sequental predcton wll be done well through a number of one-step predctons n a row. However, a predcted state s t t wll greatly devate from the dynamcally feasble regon by the accumulaton of one-step predcton errors. Therefore, t s requred to deal wth such devaton. In the real world or the system, a state s t+ always satses the robot's dynamc constrants. For example, a z- poston of each foot should be over zero and each jont's angle should be bounded. I wll denote such constraned regon by the feasble data regon (FDR). FDR s truly bounded snce I excluded some features (x w and y w ) that are not bounded n Secton II. For most one-step predctons by an ML smulator, a predcted state s t+ usually devates from FDR a lttle at least. Because the predcton for s t+ uses such s t+ as features, the predcton error wll be further ampled. Thus, we need to adjust the devated s t+ to be n FDR. However, I cannot explctly dene FDR, e.g. lettng each foot's z-poston over zero, because the goal of ths research s on the development of an ML smulator that does not consder any specc nformaton about the robot. Because the problem I deal wth s data-rch, I can collect a very large T ranp ool coverng most of FDR. If I collect nnte number of tranng data, T ranp ool wll be equal to FDR. The smplest way of utlzng T ranp ool to adjust a devated state s t+ s gvng the upperbound U and the lowerbound L to each element of the state s t+ lke the followng: s t+ := mn({u, max({s t+, L })}).

4 TABLE I COMPARISON OF NMSE FOR VARIOUS REGRESSION METHODS LR, LWPR, GPR, SOR AND LGPR Method z w x b y b z b φ θ φ θ ψ ρ ρ ρ ρ LR LWPR GPR SoR LGPR x devated predcton (s t+ ). α-proj MBR..8 q FDR nmse.6.. Fg.. tranng data ponts Comparson of two adjustng methods, α-p ROJ and Of course, the boundares are those observed n T ranp ool and ths method s called hereafter. However, there can be a tghter adjustment called α-p ROJ as shown n Fg.. Let me denote by q = mn p T ranp ool p s t+ ; the tranng data pont closest to the devated predcton s t+. Then α-p ROJ can be wrtten as the followng adjustment: s t+ := ( α)s t+ + αq,.. α I used nmse for each element as an error measurement metrc for the one-step predcton error. However, for the sequental predcton error, I used two other dfferent error metrcs: the non-cumulatve error and the cumulatve error. The non-cumulatve sequental predcton error s measured for each element as s t s t, where s t denotes the -th element of the actual state at tme t. The reason why I call t `non-cumulatve' s that every element of a state s s non-cumulatve. As I already dscussed n Secton II, each element except the z-poston z w s not cumulatve because t s orgnated from the body frame. Also, z w does not accumulate due to the gravty. Therefore, t s not possble to observe a cumulatve behavor of the sequental predcton error wth these elements of s. Instead, I computed (x w, y w ) from s and used (x w x w) + (y w yw) as the cumulatve sequental predcton error. The emprcal results for no adjustment, and α- P ROJ wth varous α, are gven n the followng secton. VII. EMPIRICAL RESULTS A. RESULTS OF ONE-STEP PREDICTION T ranp ool and T estp ool have 67 and 999 data ponts, respectvely. Computaton was performed on a laptop wth Intel Core--Duo L5.GHz CPU and GB RAM. Table I shows the comparson of performance for all regresson methods I dscussed above. Dfferent sze of tranng data set was used for each regresson method. Ths s because some of regressors (full GPR and SoR) suffer from very hgh cost for computaton and memory as the sze of tranng data ncreases. I mentoned n Secton IV that storage costs x..8 GPR SoR (Random) SoR (CV-Test) L-GPR # tranng data Fg.. nmse of varous regresson methods for predctng φ for full GPR and SoR are O(n ) and O(mn), respectvely. Therefore, full GPR couldn't work wth more than 5 tranng data due to lack of memory, and SoR couldn't hold mn over ( 5 ) for the same reason. I carefully optmzed each regressor's hyperparameters and determned the sze of tranng data set such that t yelds the best performance consderng the lmtaton of the system resource. All of 67 tranng data were used for LR, LWPR, and LGPR. 5 and tranng data were used for full GPR and SoR (where, m=), respectvely. As shown n Table I, GPRs outperformed over LR and LWPR. LWPR was poorer than expected. Among GPRs, LGPR was the best. Fg. shows nmse for predctng φ, for each GPR method, wth respect to dfferent sze of tranng data set. For smplcty, I wll not show nmse for every element of a state. Instead, only the hardest element to be predcted, the roll angle dfference φ, wll be compared. Fg. descrbes that the accuracy of most regressors roughly ncreases as the sze of the tranng data set. However, some regressors suffer from dealng wth such large tranng set. As I mentoned prevously GPR couldn't take advantage of tranng data over 5, although ts performance was the best wth tranng data less than 5. I tred to x the number of support data ponts m of SoR to be for ts best performance. However, mn couldn't be over, so t was unavodable to decrease m from to and for the case of n= and n=67. Ths s why SoR's performance got worse wth n= and n=67. For LGPR, dfferent number of clusters k was used for each case. I set the sze of tranng data n a sngle local GPR model to be approxmately, so that k s, 5,, and for each case. LGPR excelled over all other regresson methods. We can see that only LGPR can handle very large tranng data set and the accuracy s the best. Fg. also shows that SoR wth k-fold-cv test outperformed over SoR wth random selecton of m support data ponts.

5 Cumulatve Sequental Predcton Error α-proj (α=.5) Non-cumulatve Sequental Error x 8 6 α-proj (α=.5) α-proj (α=.75) α-proj (α=.) Tme Step Element Fg. 5. Cumulatve Sequental Predcton Error Cumulatve sequental error for the robot movng straght forward α-proj (α=.5) Tme Step Fg. 6. Cumulatve sequental error for the robot turnng left backward B. RESULTS OF SEQUENTIAL PREDICTION I wll compare the two adjustng methods α-p ROJ and n terms of two dfferent error measurement metrcs, the cumulatve error and the non-cumulatve error, as explaned n Secton VI. Frst, I wll take a look at the cumulatve behavor of the sequental error and how two methods releve t. Second, I wll check the non-cumulatve error and see f they actually make the sequental predcton converge to FDR. Fg. 5 and Fg. 6 show the cumulatve sequental error for LttleDog's two knds of move (F and BL, see Secton II). No adjustment gave the worst result. and α-p ROJ are ndeed helpful to reduce the sequental predcton error, but α-p ROJ, α >.5, gave the best result. I omtted the result of α =.75 and α =. because they yelded smlar result as α =.5. At last, the ML Smulator wth α-p ROJ defeats. In the -d vsualzaton of these results, α-p ROJ, α >.5 looks very smlar to the real robot moton rather than. Note that smulator's parameters are very carefully tuned to make ts moton look smlar to the real LttleDog. Fg. 7 shows the element-wse error sum for all tme steps, T t= s t s t ; ths represents the non-cumulatve sequental error. The result mples the convergence crteron for FDR. For ths specc problem, α=.5 s enough for the predcton to converge. Fg. 7. Element-wse non-cumulatve sequental error VIII. CONCLUSION I developed a smulator that learns and predcts quadruped trot gats wthout explctly consderng the dynamcs over t. I nvented the local Gaussan process regresson (LGPR) that sgncantly reduces the one-tme-step predcton error by takng advantage of a large set of tranng data. It has k locally full GPR models and predcts wth the local model closest to each test data pont. I also proposed the projecton method called α-p ROJ that can prevent the predcton over longer tme scales from dvergng. By projectng a predcted state to the feasble data regon (FDR) n whch dynamcs constrants are satsed, α-p ROJ greatly reduces the predcton error over longer tme scales. Emprcal results valdate that my approach s more efcent for smulatng quadruped trot gats than the smulator and other methods I tred. Ths research can be extended to problems wth harder motons such as gallopng and jumpng. Addng heght/slope map of a terran to a feature set wll allow the ML smulator to deal wth a bumpy terran. Local GPR actually approxmates the orgnal covarance matrx as the block dagonal one. Allowng the blocks to overlap (one data pont s belong to multple local clusters) can mprove the predcton. Mult-task learnng can also be appled to reduce the predcton error. Acknowledgement: I thank Zco Kolter for hs helpful advce on ths project. REFERENCES [] S. Vjayakumar, A. D'Souza, and S. Schaal, Incremental onlne learnng n hgh dmensons, Neural Computaton, vol. 7, no., pp. 6 6, 5. [] C. E. Rasmussen and C. K. I. Wllams, Gaussan Processes for Machne Learnng. Cambrdge, MA: The MIT Press, 6. [] J. P. Duy Nguyen-Tuong, Local gaussan process regresson for realtme model-based robot control, n IEEE/RSJ Internatonal Conference on Intellgent Robots and Systems, Sept. 8. [] M. Seeger, Some aspects of the splne smoothng approach to nonparametrc regresson curve ttng, n J. Roy. Stat. Soc., vol. B, 7(). Sprnger-Verlag, 985, pp. 5. [5] G. Wahba, G. Wahba, X. Ln, X. Ln, F. Gao, F. Gao, D. Xang, D. Xang, R. Klen, and B. Klen, The bas-varance trade-off and the randomzed gacv, n Advances n Neural Informaton Processng Systems. MIT Press, 999, pp [6] A. J. Smola and P. Bartlett, Sparse greedy gaussan process regresson, n Advances n Neural Informaton Processng Systems. MIT Press,, pp [7] J. Quñonero Candela and C. E. Rasmussen, A unfyng vew of sparse approxmate gaussan process regresson, Journal of Machne Learnng Research, vol. 6, pp , 5. [8] M. Seeger, Fast forward selecton to speed up sparse Gaussan process regresson, n Workshop on Artcal Intellgence and Statstcs 9,. 5