Fast and Efficient Incremental Learning for High-dimensional Movement Systems

Transcription

1 Vjayakumar, S, Schaal, S (2). Fast and effcent ncremental learnng for hgh-dmensonal movement systems, Internatonal Conference on Robotcs and Automaton (ICRA2). San Francsco, Aprl 2. Fast and Effcent Incremental Learnng for Hgh-dmensonal Movement Systems Sethu Vjayakumar * sethu@bran.rken.go.jp Stefan Schaal sschaal@usc.edu *Laboratory for Informaton Synthess, Rken Bran Scence Research Insttute, Wako, Satama, Japan Computer Scence and Neuroscence, HNB-3, Unv. of Southern Calforna, Los Angeles, CA Kawato Dynamc Bran Project (ERATO/JST), 2-2 Hkarda, Seka-cho, Soraku-gun, 69-2 Kyoto, Japan Abstract: We ntroduce a new algorthm, Locally Weghted Projecton Regresson (LWPR), for ncremental real-tme learnng of nonlnear functons, as partcularly useful for problems of autonomous real-tme robot control that requres nternal models of dynamcs, knematcs, or other functons. At ts core, LWPR uses locally lnear models, spanned by a small number of unvarate regressons n selected drectons n nput space, to acheve pecewse lnear functon approxmaton. The most outstandng propertes of LWPR are that t ) learns rapdly wth second order learnng methods based on ncremental tranng, ) uses statstcally sound stochastc cross valdaton to learn ) adjusts ts local weghtng kernels based on only local nformaton to avod nterference problems, v) has a computatonal complexty that s lnear n the number of nputs, and v) can deal wth a large number of possbly redundant and/or rrelevant nputs, as shown n evaluatons wth up to 5 dmensonal data sets for learnng the nverse dynamcs of an anthropomorphc robot arm. To our knowledge, ths s the frst ncremental neural network learnng method to combne all these propertes and that s well suted for complex on-lne learnng problems n robotcs. Introducton Motor control of complex movement systems requres knowledge of a varety of contnuous valued functons, for nstance coordnate transformatons of the manpulator knematcs and models of the forward or nverse dynamcs. Whenever analytcal methods are not avalable to derve these functons, e.g., as frequently the case n lghtweghted and complex (humanod) dexterous robots (e.g., Fgure ), learnng approaches need to be employed to fnd approxmate solutons. However, functon approxmaton for hgh dmensonal nonlnear motor systems remans a nontrval problem. An deal algorthm for such tasks needs to elmnate redundancy n the nput data, detect rrelevant nput dmensons, keep the computatonal complexty less than quadratc n the number of nput dmensons, and, of course, acheve accurate functon approxmaton and generalzaton. In ths paper, we suggest to accomplsh these goals wth technques of projecton regresson. The key dea of projecton regresson s to cope wth the complextes of hgh dmensonal functon approxmaton by decomposng the regresson nto a sequence of onedmensonal localzed regressons along partcular drectons n nput space. The major dffculty of projecton regresson becomes how to select effcent projectons,.e., to acheve the best fttng result wth as few as possble onedmensonal regresson. Fgure : Humanod robot n our laboratory Prevous work n the learnng lterature has focussed on fndng good global projectons for fttng nonlnear onedmensonal functons. Among the best known algorthms s projecton pursut regresson ([]), and ts generalzaton n form of Generalzed Addtve Models ([2]). Sgmodal neural networks can equally be conceved of as a method

2 of projecton regresson, n partcular when new projectons are added sequentally, e.g., as n Cascade Correlaton [3]). Here we suggest an alternatve method of projecton regresson, focussng on fndng effcent local projectons. Local projectons can be used to accomplsh local functon approxmaton n the neghborhood of a query pont. Such methods allow fttng locally smple functons, e.g., low order polynomals, along the projecton, whch greatly smplfes the functon approxmaton problem. Local projecton regresson can thus borrow most of ts statstcal propertes from the well-establshed methods of locally weghted learnng and nonparametrc regresson ([4], [5]). Counterntutve to the curse of dmensonalty ([6]), local regresson methods can work successfully n hgh dmensonal spaces ([7]), as we wll emprcally demonstrate below. In the next secton, we wll frst motvate why functon approxmaton n hgh dmensons s complcated, and why there s hope that the curse of dmensonalty s not really a problem for hgh-dmensonal movement systems. Second, we wll ntroduce our new learnng algorthm, Locally Weghted Projecton Regresson (LWPR) that can effcently deal wth hgh-dmensonal learnng problems. In the last secton, we wll show learnng results on synthetc data and data from a Sarcos Dexterous Robot, an anthropomorphc robot arm, whose nverse dynamcs model was learned by our algorthm, even when contamnated wth rrelevant and redundant nputs. 2 The Curse of Dmensonalty Both n research of bologcal and robotc motor control, the need for nternal models has been emphaszed n order to acheve accurate control of fast movements ([8]; [9]). For models wth only few nput dmensons, learnng approaches have been qute successful (e.g, []; []; [2]). However, for hgher dmensonal learnng tasks, t has been unclear whether learnng approaches can succeed. For nstance, we are nterested n learnng the nverse dynamcs model of a humanod robot (Fgure ). The robot has 3 degrees-of-freedom (DOFs), amountng to a 9-dmensonal nput space to the nverse dynamcs functon (3 poston, 3 velocty, and 3 acceleraton states). A rgd body dynamcs model performs poorly for ths hydraulcally actuated lght-weghted system. Dependng on the nonlnear actvaton functon employed n the network unts, two categores of neural networks are avalable for such a task. The tradtonal sgmodal actvaton functon or any other functon wth unbounded support tres to fnd global projectons n nput space that lead to a good approxmaton of the data. It s well known that these networks learn rather slowly n hghdmensonal spaces, and that the network structure,.e., the number of hdden unts, needs to be chosen carefully to acheve good learnng results. Moreover, neural networks wth unbounded actvaton functons are very vulnerable to nference problems,.e., the unlearnng of relevant knowledge when traned on new data ponts ([3]). Snce an autonomous robot n a dynamc envronment wll encounter new data all the tme, complex off-lne re-tranng procedures would need to be devsed to cope wth the nterference problem, and a substantal amount of data would have to be stored for the re-tranng. From a practcal pont, ths approach s hardly useful. Alternatvely, networks types can be selected that have bounded support n ther actvaton functons. Radal bass functon networks wth Gaussan kernels are among the best known n ths category ([4]; [5]). The selecton of the number of hdden unts s usually easer wth ths network type, and tranng proceeds rather fast and has reduced nterference problems. However, n a 9- dmensonal nput space, t s possble that a huge number of radal bass functons would be needed to cover the nput space: even f each nput dmenson s only covered by 2 Gaussans, an astronomcal number of 2 9 Gaussans would be requred for our humanod. Therefore, the only hope for learnng approaches s that the data generated by a movement system actually le on low dmensonal dstrbutons. We tested ths hypothess emprcally by collectng data from human and robot (Sarcos Dexterous Arm, Fgure 4) seven degree-of-freedom unconstraned arm movements by recordng about Mb of data of the jont angular trajectores durng varous movement tasks. For the human data, we assumed a bomechancally reasonable mass dstrbuton and computed the torque trajectores of every DOF; for the robot data, load sensors measured the torques drectly. Thus, we obtaned approprate data for tranng a supervsed neural network. The network we chose employed Gaussan kernels as nonlnear actvaton functon that ncluded a prncple component dmensonalty reducton (PCA) n each kernel ([7]). Importantly, the PCA automatcally determned the number of local dmensons needed n each kernel to accomplsh good functon approxmaton. Results of ncremental learnng wth ths system are shown n Fgure 2. Besdes that the neural network acheved very good approxmaton results, characterzed by a low normalzed mean squared error (c.f. [6]), t most noteworthy that the network only requred on average 4 to 6 dmensons locally for good functon fttng (cf. the dashed lnes n Fgure 2). Ths example s only to hghlght the problem of functon fttng n hgh-dmensonal spaces t wll never be possble to collect enough data wth a real robot to fll such bg spaces, even when runnng the robot for hundreds of years. 2

3 These analyses confrmed our assumpton that movement data n hgh-dmensonal spaces actually le on locally low dmensonal dstrbutons. Approprate learnng algorthms that can dscover the local dstrbutons should thus be able to approxmate really complex nternal models. a) b) nmse On Test Set nmse On Test Set.5 Local Learnng #Dmensons 9.4 Lnear Regresson #Tranng Iteratons Local Learnng Parametrc Model Lnear Regresson #Dmensons 5 #Tranng Iteratons Fgure 2: Learnng nverse dynamcs models wth a nonparametrc neural network that uses Gaussan actvaton functons wth prncpal component-based dmensonalty reducton n each Gaussan. The local learnng curves show the reducton of the mean squared error as a functon of the tranng teratons. The lnear regresson straght lne s the result of a global lnear regresson ft of the tranng data, and the Parametrc Model lne demonstrates the results from usng a rgd-body based parameter estmaton method to ft the nverse dynamcs model ([]) the latter method was only applcable for the robot data. a) Learnng from human behavoral data, b) learnng from robot data. A drawback of the PCA-based local dmensonalty reducton of ([7]), however, s that PCA only reduces the dmensonalty by lookng at the nput data. In ths way, PCA can mstake nose wth relevant nput sgnals, and rrelevant dmensons, as they may occur n some learnng problems, wll nfluence the results of learnng. These consderatons lead to a new learnng network, as descrbed n the next secton. 3 Locally Weghted Projecton Regresson In the followng, we assume that the data generatng model for our regresson problem has the standard form y = f( x)+ ε, where x R n s a n-dmensonal nput vector, the nose term has mean zero, E{ ε } =, and the output s one-dmensonal. The key concept of our regresson #Dmensons n Regresson #Dmensons n Regresson network s to approxmate nonlnear functons by means of pecewse lnear models. The regon of valdty, called a receptve feld, of each lnear model s computed from a Gaussan functon: 2 x c D x c () T w k = exp ( k ) k( k) where c k s the center of the k th lnear model, and D k corresponds to a dstance metrc that determnes the sze and shape of regon of valdty of the lnear model. Gven an nput vector x, each lnear model calculates a predcton y k. The total output of the network s the weghted mean of all lnear models: K wy k= k k ŷ = K (2) w k= k Prevous work ([3]) computed the outputs of each lnear model y k by tradtonal recursve least squares regresson over all the nput varables. Learnng n such a system, however, requred more than O(n 2 ) computatons whch became nfeasble for about more than dmensonal nput spaces. Here we suggest reducng the computatonal burden n each local lnear model by applyng a sequence of one-dmensonal regressons along selected projectons u r n nput space (note that we dropped the ndex k from now on unless t s necessary to dstngush explctly between dfferent lnear models): Intalze: y = β,z = x x For = :r T s = uz y = y+ βs z z p s The projectons u, the unvarate regresson parameters β, the mean x, and the number of projectons r are determned by the learnng algorthm. Addtonally, the learnng algorthm also fnds a projecton vector p that reduces the nput space for the next unvarate regresson. As wll be explaned below, ths step allows to fnd more effcent projectons u at subsequent unvarate regresson steps. In order to determne the open parameters n Equaton (3), the technque of partal least squares (PLS) regresson can be adapted from the statstcs lterature ([7]). The mportant ngredent of PLS s to choose projectons accordng to the correlaton of the nput data wth the output data. The followng algorthm, Locally Weghted Projecton Regresson (LWPR), uses an ncremental locally weghted verson of PLS to determne the lnear model parameters: (3) 3

4 Gven: A tranng pont ( x, y) Update the means of nputs and output: n n λw x + wx x = W n λw β + wy β = W n where W = λw + w Update the local model: Intalze: z = x, res = y β For = :r, n a) u = λ u + w zres T b) s = zu n 2 c) SS = λ SS + w s n d) SR = λ SR + w s res n e) SZ = λ SZ + w z s f) β = SR SS g) p = SZ SS h) z z sp ) res res sβ n j) MSE = λ MSE + w res 2 In the above equatons, λ [,] s a forgettng factor that determnes how much older data n the regresson parameters wll be forgotten, smlar as n recursve system dentfcaton technques ([8]). The varables SS, SR, and SZ are memory terms that enable us to do the unvarate regresson n step f) n a recursve least squares fashon,.e., a fast Newton-lke method. Step g) regresses the projecton p from the current projected data s and the current nput data z. Ths step guarantees that the next projecton of the nput data for the next unvarate regresson wll result n a u + that s orthogonal to u. Thus, for r=n, the entre nput space would be spanned by the projectons u and the regresson results would be dentcal to that of a tradtonal lnear regresson. Step j) wll be dscussed below. There are several mportant propertes n PLS. Frst, f all the nput varables are statstcally ndependent, PLS wll fnd the optmal projecton drecton u n a sngle teraton the optmal projecton drecton corresponds to the gradent of the assumed locally lnear functon to be approxmated. Second, choosng the projecton drecton from correlatng the nput and the output data n Step a) automatcally excludes rrelevant nput dmensons,.e., nputs that do not contrbute to the output. And thrd, there s no danger of numercal problems n PLS due to redundant nput dmensons as the unvarate regressons wll never be sngular. (4) The above update rule can be embedded n an ncremental learnng system that automatcally allocates new locally lnear models as needed ([3]): Intalze the LWPR wth no receptve feld (RF); For every new tranng sample (x,y): For k= to #RF: calculate the actvaton from () update accordng to (4) end; If no lnear model was actvated by more than w gen ; create a new RF wth r=2, c=x, D=D def end; end; In ths pseudo-code algorthm, w gen s a threshold that determnes when to create a new receptve feld, and D def s the ntal (usually dagonal) dstance metrc n (). The ntal number of projectons s set to r=2. The algorthm has a smple mechansm of determnng whether r should be ncreased by recursvely keepng track of the meansquared error (MSE) as a functon of the number of projectons ncluded n a local model,.e., Step j) n (4). If the MSE at the next projecton does not decrease more than a certan percentage of the prevous MSE,.e., MSE+ > φ, where φ [ ] MSE, (6) the algorthm wll stop addng new projectons to the local model. It s even possble to learn the correct parameters for the dstance metrc D n each local model. The algorthm for ths update was derved n ([3]) for normal locally lnear regresson based on an ncremental cross valdaton technque. Ths algorthm s drectly applcable to LWPR, and s strongly smplfed, as t only needs to be done n the context of unvarate regressons. Due to space lmtatons, we wll not provde the update rules n ths paper as they can be derved from ([3]). 4 Emprcal Evaluatons In order to provde a graphcal llustraton of the learnng algorthm, as a frst test, we ran LWPR on 5 nosy tranng data drawn from the synthetc two dmensonal functon: x y x y z = ( + ) max e, e, 25. e N + (,. ) (7) shown n Fgure 3a. A second test added 8 constant dmensons to the nputs and rotated ths new nput space by a random -dmensonal rotaton matrx. A thrd test added another nput dmensons to the nputs of the second 4

5 test, each havng N(,.5 2 ) Gaussan nose, thus obtanng a 2-dmensonal nput space. The learnng results wth these data sets are llustrated n Fgure 3. In all three cases, LWPR reduced the normalzed mean squared error (thck lnes) on a noseless test set rapdly n -2 epochs of tranng to less than nmse=.5, and t converged to the excellent functon approxmaton result of nmse=. after, data presentatons. Fgure 3b llustrates the reconstructon of the orgnal functon from the 2- dmensonal test an almost perfect approxmaton. The rsng thn lnes n Fgure 3c show the number of local models that LWPR allocated durng learnng. The very thn lnes at the bottom of the graph ndcate the average number of projectons that the local models allocated: the average remaned at the ntalzaton value of two projectons, as approprate for ths orgnally two dmensonal data set. In the second evaluaton, we approxmated the nverse dynamcs model of a 7-degree-of-freedom anthropomorphc robot arm (Fgure 4a) from a data set consstng of 45, data ponts, collected at Hz from the actual robot performng varous rhythmc and dscrete movement tasks (ths corresponds to 7.5 mnutes of data collecton). The nverse dynamcs model of the robot s strongly nonlnear due to a vast amount of superpostons of sne and cosne functons n the robot dynamcs. The data conssted of 2 nput dmensons: 7 jont postons, veloctes, and acceleratons. The goal of learnng was to approxmate the approprate torque command of the shoulder robot motor n response to the nput vector. To ncrease the dffculty of learnng, we added 29 rrelevant dmensons to the nputs wth N(,.5 2 ) Gaussan nose. 5, data ponts were excluded from the tranng data as a test set. Fgure 4b shows the learnng results n comparson to a global lnear regresson of the data. From the very begnnng, LWPR outperformed the global lnear regresson. Wthn about 5, tranng ponts, LWPR converged to the excellent result of nmse=.42. It employed an average of only 3.8 projectons per local model despte the fact that the nput dmensonalty was 5. Durng learnng, the number of local models ncreased by a factor of 6 from about 5 ntal models to about 325 models. Ths ncrease s due to the adjustment of the dstance metrc D n Equaton (), whch was ntalzed to form a rather large kernel. Snce ths large kernel oversmoothes the data, LWPR reduced the kernel sze, and n response more kernels needed to be allocated. In comparson to the robot learnng results n Fgure 2, t s noteworthy that LWPR requred much fewer local dmensons than the PCA-based algorthm n ([7]). Ths s due to the fact the LWPR chooses projectons much more effcently than PCA, e.g., for ndependent nput data dstrbutons, only one projecton would suffce, as mentoned before. c) nmse on Test Set a) b) z z y -.5 y #Tranng Data Ponts x x 2D-cross D-cross 2D-cross Fgure 3: a) Target and b) learned nonlnear cross functon. c) Learnng curves for 2-D, -D, and 2-D data. 5 Conclusons Ths paper presented a new learnng algorthm, Locally Weghted Projecton Regresson (LWPR), a nonlnear functon approxmaton network that s partcularly suted for problems of on-lne ncremental motor learnng. The essence of LWPR s to acheve functon approxmaton #Receptve Felds / Average #Projectons 5

6 wth pecewse lnear models by fndng effcent local projecton to reduce the dmensonalty of the nput space. Hgh-dmensonal learnng problems can thus be dealt wth effcently: updatng one projecton drecton has lnear computatonal cost n the number of nputs, and snce the algorthm accomplshes good approxmaton results wth only 3-4 projectons rrespectve of the number of nput dmensons, the overall computatonal complexty remans lnear n the nputs.. Moreover, the mechansms of LWPR to select low dmensonal projectons are capable of excludng rrelevant and redundant dmensons from the nput data. As an example, we demonstrated how LWPR leads to excellent functon approxmaton results n up to 5 dmensonal data sets, extracted from a 7 degree-of-freedom anthropomorphc robot arm. To our knowledge, ths the frst ncremental learnng system that can effcently work n hgh dmensonal spaces. b) a) nmse on Test Set Lnear Regresson LWPR #Tranng Data Ponts Fgure 4: a) Sarcos Dexterous Robot Arm; b) Learnng curve for learnng the nverse dynamcs model of the robot from a 5 dmensonal data set that ncluded 29 rrelevant dmensons. 6 Acknowledgments Ths work was made possble by Award #9732 of the Natonal Scence Foundaton, the ERATO Kawato Dynamc Bran Project funded by the Japanese Scence and Technology Cooperaton, and the ATR Human Informaton Processng Research Laboratores. 5 #Receptve Felds 7 References [] J. H. Fredman and W. Stützle, Projecton pursut regresson, Journal of the Amercan Statstcal Assocaton, Theory and Models, vol. 76, pp , 98. [2] T. J. Haste and R. J. Tbshran, Generalzed addtve models. London: Chapman and Hall, 99. [3] S. E. L. C. Fahlman, The cascade-correlaton learnng archtecture, n Advances n Neural Informaton Processng Systems II, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 99, pp [4] T. Haste and C. Loader, Local regresson: Automatc kernel carpentry, Statstcal Scence, vol. 8, pp. 2-43, 993. [5] C. G. Atkeson, A. W. Moore, and S. Schaal, Locally weghted learnng, Artfcal Intellgence Revew, vol., pp. -73, 997. [6] D. W. Scott, Multvarate Densty Estmaton. New York: Wley, 992. [7] S. Vjayakumar and S. Schaal, Local adaptve subspace regresson, Neural Processng Letters, vol. 7, pp , 998. [8] M. Kawato, Computatonal schemes and neural network models for formaton and control of multjont arm trajectory, n Neural Networks for Control, W. T. Mller I, R. S. Sutton, and P. J. Werbos, Eds. Cambrdge, MA: MIT Press, 99, pp [9] M. I. Jordan, Computatonal aspects of motor control and motor learnng, n Handbook of percepton and acton, H. Heuer and S. W. Keele, Eds. New York: Academc Press, 996. [] C. H. An, C. G. Atkeson, and J. M. Hollerbach, Model-based control of a robot manpulator. Cambrdge, MA: MIT Press, 988. [] A. H. Fagg, N. Stkoff, A. G. Barto, and J. C. Houk, Cerebellar learnng for control of a two-lnk arm n muscle space, presented at IEEE Internatonal Conference on Robotcs and Automaton (ICRA'97), Albuquerque, NM, 997. [2] M. Cannon and J. E. Slotne, Space-frequency localzed bass functon networks for nonlnear system estmaton and control, Neurocomputng, vol. 9, pp , 995. [3] S. Schaal and C. G. Atkeson, Constructve ncremental learnng from only local nformaton, Neural Computaton, vol., pp , 998. [4] J. Moody and C. Darken, Learnng wth localzed receptve felds, n Proceedngs of the 988 Connectonst Summer School, D. Touretzky, G. Hnton, and T. Sejnowsk, Eds. San Mateo, CA: Morgan Kaufmann, 988, pp [5] R. Poggo and F. Gros, Regularzaton algorthms for learnng that are equvalent to multlayer networks, Scence, vol. 247, pp , 99. [6] C. M. Bshop, Neural networks for pattern recognton. New York: Oxford Unversty Press, 995. [7] H. Wold, Soft modelng by latent varables: the nonlnear teratve partal least squares approach, n Perspectves n Probablty and Statstcs, Papers n Honour of M. S. Bartlett, J. Gan, Ed. London: Academc Press, 975, pp [8] L. Ljung and T. Söderström, Theory and practce of recursve dentfcaton: Cambrdge MIT Press,