Online Learning for Hierarchical Networks of Locally Arranged Models using a Support Vector Domain Model

Transcription

1 Proceedings of Internationa Joint Conference on Neura Networks, Orando, Forida, USA, August 12-17, 2007 Onine Learning for Hierarchica Networks of Locay Arranged Modes using a Support Vector Domain Mode Forian Hoppe and Gerad Sommer Abstract We propose two new deveopments for our supervised oca inear approximation technique, the so caed Hierarchica Network of Locay Arranged Modes. A new mode wi be presented that defines those oca regions of the input space in which inear modes are trained to approximate the target function. This mode is based on a one-cass support vector machine and heps to improve the approximation quaity. Secondy, an onine earning agorithm for our approach wi be described that can be used in appications where training data is ony avaiabe as a continuous stream of sampes. It aows to adapted a network to a function that may change over time. The success of these two deveopments is proven with three benchmark tests. I. INTRODUCTION Supervised oca (or aso caed piecewise) inear approximation (LLA) methods (such as [16], [10], [6]) use a set of inear modes to approximate a non-inear target function. These approaches foow the divided and conquer strategy to achieve good goba approximation by the right combination of good oca inear modes. These modes are trained in a way that they can ony approximate the target function in a oca region of the input space. Such a oca region is commony caed the domain of a inear mode. The different LLA approaches have to answer two basic questions: How is the domain of a oca mode defined? And, how is the input space spit up into a set of such domains? An answer to the first question conditions the possibe shapes of the oca region that is governed by a inear mode. If a domain mode is very fexibe, it is more ikey that the corresponding inear mode can approximate the target function perfecty. Otherwise more inear modes are required to divide the input space on a finer scae to reaize simiar approximation quaity. Hence, the shapes of the domains i.e. its mode strongy infuences the needed number and the possibe approximation performance of the inear modes in a network. On the other hand side, a domain mode shoud not be too compicated as this can become computationa too demanding: the needed number of parameters shoud be kept sma so that their estimation remains feasibe. So, it is the goa of LLA approaches to find a good compromise between a fexibe but sti simpe domain mode. Given such a mode, one can decide on a strategy to divide the input space in order to answer the second question. The needed process shoud meet two main goas: the achieved approximation performance of the inear modes shoud be as good and their number as sma as possibe. Since these Forian Hoppe and Gerad Sommer are with the Department of Cognitive Systems, Institute of Computer Science, Christian Abrechts University, Kie, Germany (emai: {fh,gs}@ks.informatik.uni-kie.de). two goas are strongy dependent on each other, again a good compromise has to be estabished to fufi both. II. STATE OF THE ART LLA TECHNIQUES The k-nearest neighbor (k-nn) approach (e.g. [4]) is a straightforward reaization of the oca inear mode idea. It soves for each input x R n the east squares probem: k ( ) 2, min y i α(x) β(x)x i (1) α(x),β(x) i=1 where the training sampes (x i, y i ) with i = 1,..., k are the k-nearest neighbors to the input x. The set of nearest neighbors is determined by comparing the distances between the input x and a sampes from the training set w.r.t. some metric (typicay, the Eucidean distance). The estimated output ŷ R m is consequenty given by ŷ = α(x) + β(x)x. Note, that the k-nn approach is non-parametric and has no expicit domain mode since it uses the whoe training set to compute an output vaue. Hence it is very inefficient in both memory and computationa costs. In contrast to that, approaches ike [16], [10], [6] can be formaized as: M ŷ(x) = g k (x)βk T x, (2) k=1 where M is the number of used inear modes with their coefficient β k R n+1 and x = (x, 1) T R n+1 is the extended input x that aows a constant term in the inear equation. With the weighting function g k ( ) R the domain of the k-th inear mode is defined. This definition is specific to the different approaches. Their common idea is that depending on the distance of the input x to the domain, the weighting factors are arger or smaer and hence, wi weight the output βk T x of the inear modes differenty. The typica definition of the weighting functions g k ( ) is based on a radia basis function (RBF). The resuting domains have the shape of hyper-eipsoids in the input space. Depending on the approach specific definition of the weighting functions these hyper-eipsoids have different degrees of freedom. E.g. in [10] the main axis of the hyper-eipsoids are restricted to be parae to the axis of the input space, whie in [6] these can be oriented in any direction. Besides differences in the definition of the domain mode the approaches vary more importanty in their strategies to spit up the input space. In [10] the position and size of the domains are changed with Hebbian adaptation steps and by a gradient decent approach to minimize the east squares error function over the training set. The authors of [16] prefer to X/07/$ IEEE

2 minimize a so-caed ocay weighted error function which emphasizes that training sampes ony effect the domain s parameter they beong to. Our own offine earning agorithm (introduced in [5] and extended in [6], [7]) is a recursive scheme that divides by means of a specia custering process the input space into singe domains. This divide and conquer strategy is repeated unti the required approximation quaity is achieved in each domain or no more data sampes are avaiabe in a oca region to divide it furthermore. In the foowing, we wi present two new deveopments in our LLA the so caed Hierarchica Network of Locay Arranged Modes (HLAM) approach. A new domain mode is described that aows more fexibe shapes than those of a hyper-eipsoid. More important an onine earning agorithm for our network was reaized. It is designed to be used in an appication where training data is ony avaiabe as a continuous stream of sampes. The idea is that the input space of the target function is being expored during the runtime of the impemented system. Furthermore a HLAM shoud capture dynamic changes of the target function. This is different to the offine earning scenario where a certain input vaue aways corresponds to a certain output vaue. The onine earning agorithm can adapted a HLAM to a function that may change over time. III. NEW DEVELOPMENTS FOR HIERARCHICAL NETWORKS OF LOCALLY ARRANGED MODELS The basic definition of a HLAM is given with (2). The weighting functions are defined as: { 1 : k = argi max a g k (x) = i (x), (3) 0 : ese where a i ( ) with i = 1,..., M are so caed activation functions of the M inear modes (their definition wi be given beow). So, in the HLAM approach ony one inear mode is excusivey seected to compute the output of the whoe network. Ony the output of the inear mode with maxima activation vaue wi be weighted with one, a the other weights wi be set to zero. This excusive seection stands in contrast to the other LLA approaches which mix the output of different modes. Our definition promotes that the inear modes are stricty speciaized for their domain. This is supported by our earning agorithms as these train the inear modes ony with the data from their own domains. This parameter estimation is done individuay for each inear mode with a standard east squares method (e.g. [4]). Other approaches determine the parameters of the inear modes with a avaiabe data using weighted east squares schemes. The Support Vector Domain Mode The definition of the activation functions are based on a specia distance function that defines the shape of a domain. In [6] these functions were basicay radia basis functions defining domains shaped ike hyper-eipsoids. As an aternative we propose to use a one-cass support vector machine (SVM) to define the distant function d k : R n R as: (a) Fig. 1. Exampe of a support vector domain. Pane (a) shows a set of data sampes and the domain boundary reaized with a radia basis SVD. These sampes that are support vectors are marked with circes. Pane (b) shows the corresponding activation function. (b) S k d k (x) = α k,i K(s k,i, x), (4) i=1 where S k N is the number of the support vectors s k,i R n with their weights α k,i R and K : R n R n R is a kerne function. The user can choose any kerne function. Popuar options are: K(x, y) = (1 + x T y) d, (poynomia) K(x, y) = exp( 1 σ x y 2 ), (radia basis) (5) K(x, y) = tanh(κ 1 x T y + κ 2 ). (sigmoida) Given the function d k ( ) of the k-th domain, its activation functions is defined with: { dk (x) : d a k (x) = k (x) γ k, (6) : ese where the threshod γ k is a scaar parameter. The activation vaue is equa to the vaue of (4) if the threshod is met or exceeded. This draws a sharp boundary around the domain which stricty separates the domain s inside from its outside. Figure 1 shows an exampe of such a support vector domain (SVD). In the foowing a the parameters of the k-th domain wi be denoted with Φ k = (s k,1,..., s k,sk, α k,1,, α k,sk, γ k ). The support vectors and the weights of a domain are determined with a standard method for one-cass support vector machines (see e.g. [14]) for a set of input sampes {x i }. This set is seected by the HLAM earning agorithm which ensures that the sampes beong to a oca region of the input space. Given the support vectors and their weights the threshod γ k is set to the minimum vaue of the function d k ( ) of a support vectors s k,i, i.e. γ k = min sk,i d(s k,i ). This estabishes a domain boundary that encoses a the training data. Note, that the user has to specify additiona parameters (the reguation parameter ν and e.g. for a radia basis kerne its width σ ) for the empoyed SVM earning agorithm and the chosen kerne function. These parameters have to be manuay optimized for a specific data set. The threshods γ k have two purposes in the HLAM approach. On one hand side, it aows to reaize an automatic outier rejection mechanism. A data sampe with input vaues that are atypica for a specific appication can be detected by simpy testing if the sampe beongs to any domain.

3 If not, an exception can be thrown by the program that the input acquisition method produced data that can not be handed by the trained network. This mechanism increases the reiabiity of a system that empoys a HLAM. Standard machine earning technique do not offer this feature as they aways cacuate some output regardess how invaid the input was. If such an outier rejection mechanism is not needed a threshods γ k shoud be set to minus infinity. Then this oca mode wi compute the output which domain is cosest to the input, regardess if this beongs to the domain or not. This kind of nearest neighbor seection suits we the basic idea of the HLAM approach that assigns inear modes to oca regions of the domain space. On the other hand, the sharp domain boundary defined by the threshod γ can be expoited for a novety detection. It depends on the view point i.e. on the task at hand if a data sampe shoud be regarded as an invaid input or if the same sampe is taken as a new piece of information. In the first case the sampe shoud not be processed as it may cause harmfu effects. In the atter it coud be used to extend a trained HLAM. This idea is utiized in the new onine earning agorithm. The advantage of the new support vector domain mode is the great fexibiity of its boundary. In contrast to the former hyper-eiptica domain, the SVD mode can encose non-convex regions of the input space because the kerne functions can induce compex non-inear boundaries. Hence a given set of training sampes can be more tighty encosed. This ensures that ony those regions of the input space beong to domains that reay contain data sampes. The positive effect is that the corresponding inear modes are ony responsibe for those oca regions where they received training data. Hence they can be better speciaized to their domains. That shoud improve the approximation performance of the whoe network. Furthermore, data sampes which may be widespread in the domain space can be adequatey encosed by one singe support vector domain and hence approximated by one inear mode. Since in such a case more data is avaiabe to train a inear mode its approximation performance shoud be improved. Furthermore, with arger domains HLAMs become smaer. If more data can be assigned to the singe domains, their number wi be reduced. The Onine Learning Agorithm Two basic ideas ground the deveopment of the onine earning agorithm. The activation function of a domain defines the responsibiity of the corresponding inear mode to a given training sampe. This information is used to decide which inear mode shoud be adapted to the new training sampe. On the other hand, a so caed neighborhood graph is used to divide the domain space into oca regions w.r.t. estabished domains. This is important to decide where new domains shoud be estabished. The pseudo code is given in Agorithm 1. It defines the procedure UpdateHLAM that shoud be caed when a new sampe (x, y) is avaiabe. The agorithm works as foows: A HLAM is initiaized with no oca mode at a. The agorithm has a meta- Agorithm 1: Pseudo code of onine earning agorithm. Function UpdateHLAM Input : (x, y), M t = {(βk t, Φt k, T k t )}, Bt Output: M t+1 = {(β t+1 k, Φ t+1 k, T t+1 k )}, B t+1 begin if M = then B t+1 B t {(x, y)} if B t+1 minsampes then (β t+1, Φ t+1 ) TrainMode(B t+1 ) M t+1 = (β t+1, Φ t+1, B t+1 ) B t+1 = ese Err min k L( x T βk t, y) if Err maxerror then arg k max d k (x) (β t+1, Φ t+1, T t+1 ) UpdateM(T t, (x, y)) ese arg k max a k (x) if d (x) γ k then (β t+1, Φ t+1, T t+1 ) UpdateM(T t, (x, y)) B t+1 B t {(x, y)} G GetNeighborhoodGraph(B t+1, {Φ t k }) (, s) arg i,j max G i,j if G,s minsampes then T t+1 GetSampes(B t+1,, s) B t+1 RemoveSampes(B t+1,, s) (β t+1, Φ t+1 ) TrainMode (T t+1 ) M t+1 M t (β t+1, Φ t+1, T t+1 ) end Function UpdateMode Input : T t, (x, y) Output: (β t+1, Φ t+1, T t+1 ) begin T t+1 T t {(x, y)} if T t+1 > maxbuffersize then T t+1 RemoveOdestSampe(T t+1 ) (β t+1, Φ t+1 ) TrainMode (T t+1 ) end Function GetNeighborhoodGraph Input : T = {(x j, y j )}, {Φ }, Output: G N M M begin G 0 fora j do k 1 arg min x j 1 S S i=1 s,i k 2 arg { k1} min x j 1 S S i=1 s,i G k1,k 2 G k1,k G k2,k 1 G k2,k end

4 TABLE I RESULTS OF THE MACKEY-GLASS BENCHMARK TEST (SEE TEXT FOR AN EXPLANATION ABOUT THE HLAM MARKED WITH A STAR). Method Number of RMSE Hidden Units Train Set Test Test HALM offine HLAM onine HLAM offine* GMN [10] RBF-AFS [2] OLS [1] parameter minsampes that specifies how many sampes at east have to be assigned to one inear mode. After initiaization, this user chosen number of sampes have to be coected before the first inear mode and its domain wi be added to the HLAM. Therefore new sampes are temporary stored in a buffer T t. Throughout the runtime of the system, this buffer wi contain a the sampes that coud not be assigned to a inear mode of the HLAM. If the size of buffer T t equas minsampes, the first mode and its domain wi be estabished with a the sampe from the buffer. The mode s parameters β t are estimated with a standard east squares method, whie the domain s parameter Φ t are determined as described above. In the pseudo code this both is encapsuated in the procedure TrainMode. Aong with β t and Φ t an extra buffer T t containing a the used training sampes is added to the HLAM. This buffer T t represents the sampe history of each oca mode. How it works in detai wi be expained down beow. After the first mode was added to a HLAM, for each new training sampe it wi be decided if an aready existing inear mode shoud be updated or a new mode shoud be added to the HLAM. The criterion for this decision is if the sampe coud be successfuy approximated or not. Therefore the minima oss L( x T βk t, y) of a inear modes of a HLAM is determined and compared with a user chosen threshod maxerror. This oss function can be seected as appropriate. We use: L(ŷ, y) = (ŷ y) 2. If the approximation performance is good enough, the new sampe is used to update one inear mode. Before the question which mode shoud be updated can be setted one shoud first discuss how this is done: As noted above, the onine agorithm stores for each inear mode the history of sampes that were assigned to the mode during runtime. The sampes are coected in a buffer T t which serves as a singe training set. The update process first adds the new sampe to T t and then trains the inear mode and its domain with T t. Obviousy, the usabiity of the agorithm woud be quite imited if the size of the buffer is not. The computationa costs in time and memory coud easiy become exhausting. The buffer coud grow arbitrariy as ong as the system keeps running and coecting new sampes. As a resut of, the retraining of the inear modes and domains woud become too time demanding. Beside such practica probems, a HLAM coud not adapted to a dynamicay changing target function. No sampe of an input-output correspondence woud be forgotten. Over time, the inear modes woud have to sove the impossibe task to cope with data that contradicts itsef. Hence the size of the buffer T t is imited by another metaparameter caed maxbuffersize. The access to the buffer has to be impemented ike a queue, so that the odest sampe wi be discarded if a new sampe is avaiabe and the buffer s maxima size is reached. How this can be done exacty is eft to a programmer. In the pseudo code, it keeps hidden behind the procedure RemoveOdestSampe. The parameter maxbuffersize has to be chosen by the user in accordance with the assumptions about the dynamics of the target function and the avaiabe memory and CPU power. The buffer shoud be sma if the target function is quicky changing and the computationa burden has to be kept ow. In any case maxbuffersize must be arger or equa to the other meta-parameter minsampes. So, the question remains which inear mode shoud be updated by this process. The basic idea is to adapteth this mode to the new sampe which domain contains or is the nearest to it. Therefore the agorithm seects the mode that has the highest vaue of d k (x) defined with (4). It is important to use the function d k ( ) instead of the proper activation function a k ( ) because these drop to minus infinity at the boundary of a domain. Hence those domains, that do not contain the new sampe, have the same activation vaue. So, it can not be decided which domain is the nearest to the sampe. Ony domains that encose the new sampe, hence overap each other, coud be discriminated with their activation function vaue. But if ony such domains woud be taken into account for an update, domains coud never expand, instead their size woud shrink over time. The reason for that is grounded in the update process with the history buffer. The size of a domain is determined by the sampes that are ocated at the boundary of the domain. But these wi eventuay be pushed out of the history buffer by sampes that must ay inside of the domain to be accepted for an update. Hence the new sampes from within the former domain wi define the new but tightened boundary. In contrast to that, a domain can expand if sampes are used for an update that are in the vicinity and not necessariy inside of it. This conception is impemented by the seection mechanism that updates this mode that has the highest vaue of d k (x). If the approximation performance of a HLAM is not satisfying for the new sampe, the onine agorithm improves the network by foowing two ideas. If a inear mode aready exists that shoud hande the new sampe, this mode wi be updated with it. Additionay, the structure of the network may be changed by inserting a new inear mode. The first idea is impemented in a straightforward manner taking advantage of the sharp boundaries of the SVD mode: The mode with the highest activation vaue a k (x) is checked if its domain contains the new sampe, and updated in such a case. The second idea needs more effort to be impemented. It reies essentiay on the so caed Neighborhood Graph G which was proposed in [7]. The graph G is given as a symmetric N M M matrix and expresses which domains of a

5 TABLE II RESULTS OF THE ABALONE BENCHMARK TEST. Method Number of RMSE Hidden Units Train Set Test Set HLAM offine HALM onine RANEKF [9] GAP-RBF [8] MRAN [17] RAN [12] MRAN-OLS [11] HLAM are adjacent to each other. The graph offers the advantage that a set of sampes can be grouped w.r.t. aready estabished domains of the HLAM. It custers sampes together that have the same domains as their nearest neighbors, hence are ocated in the same region of the domain space. Given G, the agorithm inserts new domains in those regions where enough sampes coud be coected. Therefore every new sampe that is not approximated good enough is first stored in the extra buffer B t. Then the neighborhood graph G is computed for these sampes. The components of G represent the number of sampes ocated in the neighborhood of these domains that correspond to the indices of the component. So, if there exists a component G,s that is equa or arger than the threshod minsampes, a new inear mode is created with a domain that is ocated in the vicinity of the -th and s-th domain. Therefore these sampes beonging to this oca region are removed from the buffer B t and used as a training set for the new inear mode and its domain. Again, to simpify the pseudo code the definitions of the procedures GetSampes and RemoveSampes are omitted. They ony have to manage the sampes with an indexing system in correspondence to the domains. One unsoved probem with this buffer is that in theory it may grow unboundedy. Sampes wi ony be removed from the buffer if enough are avaiabe in the vicinity of two domains. Hence sampes may never be used to estabish a new oca mode, hence their information wi be ost. The chances that the unused sampes are in this sense bady distributed between domains grow with the number of oca modes. In the worst case the buffer woud contain 1 2 M(M 1)(minSampe 1) sampes since 1 2M(M 1) is the number of possibe pairwise combinations of M domains. In [7] we proposed a method to fuse domains of a HLAM in order to reduce its number of inear modes. Athough it was not designed for an onine earning scenario, it is ready to be appied in it. The fusion agorithm can be started in reguar intervas after an appropriate number of sampes were received. Essentiay, it tests if one inear mode can be trained to approximate the unified training data of two adjacent domains. If this is the case, the domains wi be fused and the new inear mode wi repace the two former ones. The modifications of the agorithm to the new SVD mode are so straightforward that they can be omitted here. IV. BENCHMARK EXPERIMENTS In order to compare the new deveopments with other techniques three benchmark tests were performed. TABLE III RESULTS OF THE AUTO-MPG BENCHMARK TEST. FOR THE 50 TRIALS THE MEAN AND STANDARD DEVIATION OF THE RESULTS ARE GIVEN. Method Mean Number Mean RMSE of Hidden Units Train Set Test Set HLAM offine 2.00 ± ± ±.008 HLAM onine 8.88 ± ± ±.014 MRAN [17] 4.46 ± ± ±.023 RANEKF [9] 5.14 ± ± ±.029 GAP-RBF[8] 3.12 ± ± ±.027 MRAN-OLS [11] 2.10 ± ± ±.011 RAN [12] 4.44 ± ± ±.092 Mackey-Gass Data Set: In [10] a benchmark test with the chaotic Mackey-Gass differentia equation was described. The experiment stems originay from [13] and defines the task to predict a vaue of the time series x(t 1) = (1 a)x(t) + bx(t τ) 1 + x 10 (t τ), (7) with the parameters chosen to be: a = 0.1, b = 0.2, τ = 17 and the initia condition as x(0) = 1.2. The function f : R 4 R that has to be approximated is defined as ( ) x(t + 6) = f x(t), x(t 6), x(t 12), x(t 18). (8) For a comparison of different machine earning techniques two separate sets of sampes of f were generated. One set with t = 124,..., 1123 serves as a training set for the machine earning technique, whie the other one with t = 1124,..., 2213 is used to cacuate the RMSE as the fina test error. Abaone Data Set: The goa of this benchmark test is to estimate the age of an abaone given seven continuous and one discrete attributes. Instead of cutting the she, staining it and counting the number of rings through a microscope the age shoud be predicted with attributes that are easier to obtain. Therefore, the sex, ength, diameter, height and the weight of four different parts of an abaone are avaiabe as input vaues in the database [3] of 4177 sampes. The target output vaue is discrete and ranges between 1 and 29 years. As described in [11] the input and output vaues were normaized to the range [0.1, 0.7]. For the training 3000 sampes were randomy seected from the whoe data set. The RMSE was cacuated for the remaining 1177 sampe. Auto-Mpg Data Set: In [15] the so caed Auto-Mpg probem was stated to predict the city-cyce fue consumption in mies per gaon in terms of three mutivaued discrete and four continuous attributes. The data set that is avaiabe at [3] contains 398 sampes of this continuous target function. As mutivaued discrete input vaues the number of cyinder, the mode year and the origin are given. The continuous attributes are the dispacement, horsepower, weight and acceeration of the cars. The benchmark test was repeated as described in [11]. Therefore the input and output vaues were normaized to the range [0, 1]. Since the number of sampes is quite imited, a number of 50 trias were performed with different training and test sets. For each tria, 320 sampes were randomy chosen for training, whie the remaining 78 sampes were

6 TABLE IV VALUES OF META-PARAMETERS OF THE HLAMS USED FOR THE Offine Onine BENCHMARK TESTS. Meta-Parameters Mackey-Gass Abaone Auto-Mpg minsampes maxerror σ ν minsampes maxbuffersize maxerror σ ν used to compute the RMSE. The mean vaue of the trias RMSE are isted as the fina resut of this benchmark test. Resuts: Our approach with the off- and onine earning agorithm was compared with the above described benchmark tests to the resuts reported in [10] and [11]. A resuts are isted in Tabe I, II and III. As kerne functions for the SVD mode we used radia basis functions. The vaues of the meta-parameters for the earning agorithms are given in Tabe IV. They were seected by a process of repeated training and testing. Therefore, different vaues for minsampes, the error threshod maxerror and maxbuffersize (ony for the onine agorithm) were chosen from an appropriate interva to train a HLAM 1. The achieved RMSEs on the test set were compared and the best one is cited as the fina resut. The tabes show the superiority of the HLAM approach over the other methods w.r.t. the achieved RMSE in a benchmark tests. Both, the off- and the onine earning agorithm performed very we. Especiay in the Mackey- Gass benchmark test, the offine HLAM reaizes a ceary better resut than the aternatives. In every case our offine earning agorithm outperforms the proposed onine version. So, the genera expectation that an offine earning scenario is easier to hande with a machine earning technique is met. It can be pointed out that the new HLAM onine earning agorithm exceeds the approximation quaity of such offine agorithms ike e.g. RANEKF or MRAN. To iustrate the point that the fexibiity of the SVD mode heps to reduce the number of needed inear modes, one HLAM was speciay trained for the Mackey-Gass benchmark test (marked with a star in Tabe I). With the meta-parameter minsampes set to 20, the achieved RMSE degraded sighty (athough sti better than the competitors) but the number of modes coud be reduced by more than 50 %. V. CONCLUSION Aong with new benchmark tests, we presented two new deveopments for our Hierarchica Network of Locay Arranged Modes. A domain mode that is based on a one-cass support vector machine was proposed as an aternative to the former hyper-eiptica one. It features a more fexibe domain boundary that heps to improve the approximation quaity 1 Athough in the Auto-Mpg benchmark test 50 HLAMs have to be trained for 50 different training and test sets the meta-parameters were ony optimized once. of a HLAM and to reduce the number of needed inear modes. On the other hand, an onine earning agorithm for the HLAM approach was deineated. It is designed to be used in an appication where training data is ony avaiabe as a continuous stream of sampes. It aows to adapted a HLAM to a function that may change over time. Consistent with the basic HLAM idea, the onine earning agorithm promotes speciaization of the inear modes to oca regions of the domain space. The success of these two deveopments coud be proven with three benchmark tests. ACKNOWLEDGMENTS: The work presented here was supported by the the European Union, grant COSPAL (IST ). However, this paper does not necessariy represent the opinion of the European Community, and the European Community is not responsibe for any use which may be made of its contents. REFERENCES [1] S. Chen, C. F. N. Cowan, and P. M. Grant. Orthogona east squares earning agorithm for radia basis function networks. IEEE Transactions on Neura Networks, 2: , [2] K. B. Cho and B. H. Wang. RBF based adaptive fuzzy systems and their appications to system identification and prediction. Fuzzy Sets and Systems, 83: , [3] C.L. Bake D.J. Newman, S. Hettich and C.J. Merz. UCI repository of machine earning databases, [4] T. Hastie, R. Tibshirani, and J. Friedman. The Eements of Statistica Learning. Springer Series in Statistics. Springer, [5] F. Hoppe and G. Sommer. Loca inear modes for system contro. In Proceedings of Int. Conf. on Neura Information Processing (ICONIP), pages , [6] F. Hoppe and G. Sommer. Ensembe earning for hierarchies of ocay arranged modes. In Proceedings of IEEE Word Concress on Computationa Inteigence, pages , [7] F. Hoppe and G. Sommer. Fusion agorithm for ocay arranged inear modes. In 18th Internationa Conference on Pattern Recognition (ICPR 06), pages III: , [8] Guang-Bin Huang, Paramasivan Saratchandran, and Narasimhan Sundararajan. An efficient sequentia earning agorithm for growing and pruning RBF (GAP-RBF) networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 34(6): , [9] V. Kadirkamanathan and M. Niranjan. A function estimation approach to sequentia earningwith neura networks. Neura Computation, 5(6): , [10] Loo Chu Kiong, Mandava Rajeswari, and M. V. C. Rao. Extrapoation detection and novety-based node insertion for sequentia growing muti-experts network. Neura Network Word, 2: , [11] Xiaoping Lai and Bin Li. An efficient earning agorithm generating sma rbf neura networks. Neura Network Word, 15: , [12] J. Patt. A resource-aocating network for function interpoation. Neura Computation, 3(2): , [13] J. Patt. A resource-aocating network for function interpoation. Neura Computation, 3(2): , [14] John C. Patt, Bernhard Schökopf, John Shawe-Tayor, Aex J. Smoa, and Robert C. Wiiamson. Estimating the support of a highdimensiona distribution. Technica Report MSR-TR-99-87, Microsoft Research (MSR), November An abridge version of this document wi appear in Neura Computation. [15] J. Ross Quinan. Combining instance-based and mode-based earning. In ICML, pages , [16] Stefan Schaa and Christopher G. Atkeson. Constructive incrementa earning from ony oca information. Neura Computation, 10(8): , [17] Lu Yingwei, N. Sundararajan, and P. Saratchandran. A sequentia earning scheme for function approximation using minima radia basis function neura networks. Neura Computation, 9(2): , 1997.