A New Algorithm for Autoregression Moving Average Model Parameter Estimation Using Group Method of Data Handling

Transcription

1 Annals of Biomedical Engineering, Vol. 29, pp , 2001 Printed in the USA. All rights reserved /2001/29 1 /92/7/$15.00 Copyright 2001 Biomedical Engineering Society A New Algorithm for Autoregression Moving Average Model Parameter Estimation Using Group Method of Data Handling KI H. CHON and SHENG LU Department of Electrical Engineering and Center for Biomedical Engineering, City College of the City University of New York, New York, NY (Received 24 February 2000; accepted 2 November 2000) Abstract A new algorithm for autoregresive moving average ARMA parameter estimation is introduced. The algorithm is based on the group method of data handling GMDH first introduced by the Russian cyberneticist, A. G. Ivakhnenko, for solving high-order regression polynomials. The GMDH is heuristic in nature and self-organizes into a model of optimal complexity without any a priori knowledge about the system s inner workings. We modified the GMDH algorithm to solve for ARMA model parameters. Computer simulations have been performed to examine the efficacy of the GMDH and comparison of the GMDH is made to one of the most accurate and one of the most widely used algorithms, the fast orthogonal search FOS and the least-squares methods, respectively. The results show that in some cases with noise contamination and incorrect model order assumptions, the GMDH performs better than either the FOS or the least-squares methods in providing only the parameters that are associated with the true model terms Biomedical Engineering Society. DOI: / Keywords ARMA model, Group method of data handling, Parameter estimation, Dynamic noise, Additive noise, MA model, AR model, Model order selection. INTRODUCTION Address correspondence to Ki H. Chon, City College of NY, Dept. of Electrical Eng., Steinman Hall, Rm. 677, Convent Ave. at 138th Street, NY, NY Electronic mail: kichon@e .engr.ccny.cuny.edu 92 The ubiquity of autoregressive moving average ARMA models in physiological system identification is owed to their having been successfully demonstrated in recent years. 3 5,10,11 Their use has been greatly aided by advances in overcoming the holy-grail problem of model order determination during ARMA model parameter estimation. 4,10,11 One of the most notable and accurate algorithms introduced to determine model order selection efficiently is the novel work by Korenberg. 10,11 Korenberg s fast orthogonal search FOS algorithm has been shown to be robust in most cases in obtaining correct ARMA model parameters despite incorrect model order selection. This observation remains valid even with significant noise added to the system response. 4,10,11 The FOS algorithm which relies on the sequential search procedure to extract only the significant ARMA model terms, however, is suboptimal. It is suboptimal because the FOS does not test for the minimum error across all possible subsets of ARMA model candidate functions within the candidate function space. Thus, there are cases when the FOS is not able to identify correctly the true ARMA model parameters even with no noise present. To alleviate some of the aforementioned shortcomings of the FOS, we introduce a new algorithm which searches for all possible ARMA model candidate functions although not a global search of candidate functions and, thus, in the simulation examples considered, provides accurate estimation of the system parameters. The algorithm we present is a modification of the algorithm introduced by the Russian mathematician and cyberneticist, A. G. Ivakhnenko, in the mid 60s. The algorithm was used for estimating higher-order regression polynomials and is denoted the group method of data handling GMDH. The GMDH was designed to construct successively higher-order regression equations at each iteration from equations of the previous iteration while retaining only those that best approximate the given data set. In this way a high-order regression type model is evolved guided only by survival of the fittest. The main idea of the GMDH is to have the algorithm construct a model of optimal complexity based only on data; a self-organizing model that can be used to solve prediction, identification, control synthesis, and other system problems. 6 9 The GMDH algorithm is similar to the genetic algorithm approach to parameter estimation, in which the true candidate terms are created based on the principle of survival of the fittest. Note that in some genetic algorithms, only the fittest offspring survive based on the rank selection scheme. 2,13 The aim of the present study is to show that the GMDH, originally designed for estimating higher-order regression polynomials, can be modified to be one of the better algorithms for ARMA model parameter estima-

2 ARMA MODEL PARAMETER ESTIMATION 93 tion. To showcase the efficacy of the GMDH, comparisons between the GMDH, the FOS, and the least-squares methods are made using simulation examples. The FOS is chosen since it is one of the most accurate algorithms available for ARMA parameter estimation. Least squares using the Akaike information criteria for model order selection is chosen for comparison since it is still one of the most widely utilized methods for ARMA model parameter estimation, especially in the biomedical engineering community. METHODS GMDH Algorithm We describe a modification of the GMDH algorithm to estimate the parameters of the ARMA model, in this section. The GMDH was originally formulated to solve for higher-order regression polynomials and not the difference equation upon which ARMA models are based. Consider a time-invariant autoregressive moving average process of the form P y n i 1 Q a i y n i b j x n j, j 0 where P and Q represent the maximum autoregressive AR and moving average MA model orders, respectively. Note that Eq. 1 is in a control format rather than in the classical format known to statisticians. The parameters a(i) and b( j) represent to-be-estimated coefficients of the AR and MA terms, respectively. The basic steps of the newly developed algorithm to calculate ARMA model parameters are as follows. Step (1). Partition the data with the first nt rows designated as the training set and the remaining rows as the testing set. From the training set form a matrix of nt observations of ARMA model terms like the ones shown below. Note that the first nt rows and the last N nt denote training and testing data, respectively. The testing data set is only used in step 3 below, to measure the mean-square error of the model identified with the training data set 1 y 0 y 1 y 1 P x 1 x 0 x 1 Q y 1 training y 2 y 1 y 0 y 2 P x 2 x 1 x 2 Q data ] y nt testing ] data y N ] ] ] ] ] ] y nt 1 y nt 2 y nt P x nt x nt 1 x nt Q ] ] ] ] ] ] ] ] y N 1 y N 2 y N P x N x N 1 x N Q Matrix R. Y R 1 R 2 R m Step (2). Take all the variables m number of variables in the columns of the matrix R y(n 1),..,y(n P),x(n),x(n 1),x(n Q) two columns at a time and for each of these m(m 1)/2 combinations find the least-squares regression that best fits the observation vector y s. For example, for the first two combinations vectors (R 1,R 2 ) in matrix R, evaluate the least-squares polynomial of the form y A BR 1 CR 2 DR 1 2 ER 2 2 FR 1 R 2 at the data points y(0),y( 1), y(1),y(0),..., y(n 1),y(N 2) and store these N values in the first column of a new array Z. The remaining m(m 1)/2 1 columns are constructed in a similar manner. The array Z contains the new variables, some of which replace the original variables. The objective is to retain those z s that best estimate the output vector y and discard the insignificant variables. Step (3). To determine which columns of Z new variables replace the old variables in the matrix R, compute the least squares error, d j : d 2 j 1/ N nt N i nt 1 y i z ij j 1,2,..., 2 N 1/ N nt i nt 1 y i y 2, m 2 2 and order the columns of Z according to increasing least square error. Note that z ij in Eq. 2 represents an element of array Z, and y denotes the mean of the output signal y. For example, in Eq. 3, an element z 11 represents the output of a polynomial function evaluated in step 2 at the data points y(0),y( 2). One can place a restriction as to some prescribed number of new variables to replace the old variables in the matrix R, i.e.,

3 94 K. H. CHON and S. LU d j M, where M is some prescribed number. Thus, if a priori we choose P 10 and Q 10 terms and find that only the combination of the vectors y(n 1),y(n 3), y(n 1),y(n 5), and y(n 2),x(n) terms meet the leastsquares error criteria, then the new variables Z will have the following form: z 11 y 0,y 2 y 0,y 4 y 1,x 1 ] ] ] ]. 3 z N1 y N 1,y N 3 y N 1,y N 5 y N 2,x N Z 1 Z 2 Z 3 Step (4). From step 3 take the smallest of the d j.if the value of d j is smaller than the previous d j in the first iteration we assume this to be true go back and repeat steps 2 and 3. If the value of d j is greater than the previous d j we stop the process. It is our observation, however, that the value of d j becomes smaller than the previous d j as the number of iterations is increased. It is our experience that performing more iterations results in smaller mean-square error but at the expense of an increase in the number of model terms. Based on our simulations, we have determined that three iterations are sufficient to obtain accurate parameter estimates. Figure 1 illustrates the iterative process involved in the GMDH algorithm. The layers in Fig. 1 correspond to the number of iterations; for three iterations, we have three layers. Note that the first and second nodes in the first layer of Fig. 1 may represent terms y(n 1) and y(n 2), and y(n 1) and y(n 3), respectively, if these terms meet the criteria described in step 3. With increasing iteration, the number of layers will increase and the number of terms in the nodes at that layer may increase drastically, provided that terms pertaining to those nodes meet the criteria in step 3. Note that in the third layer, the nodes are ordered according to increasing least square error, i.e., node one is more significant than node two and so on. Step (5). The GMDH algorithm calls for the use of the least squares to find the coefficients of the retained variables. However, we have made a modification to the original GMDH algorithm at this step. Instead of computing the least squares of all the retained variables or all of the nodes in layer three, we compute the contribution of each of the nodes to reducing the mean-square error. If we observe either negligible decrease or increase in the mean-square error by adding an additional node, then those nodes are dropped from the model. Only those nodes that reduce the mean-square error are retained, and the coefficients pertaining to the retained nodes are estimated using the least-squares algorithm. Thus, the distinct terms from all the retained nodes are included in the ARMA model. The Simulation Results section discusses this step further see Fig. 2. The distinct advantage of the GMDH is more apparent when the number of variables and the degree of the polynomials are large. For example, with least squares, to find a regression polynomial with 4 variables of degree 4, 70 linear equations with 70 unknowns must be solved. This is not a daunting task with the present computing power, however, the equations are often illconditioned and consequently cannot be solved. The GMDH polynomials, however, can be obtained by solving far fewer linear systems of order six. The reason the GMDH requires fewer computations is that it retains only the information that is highly correlated with the testing data set. Consequently, this allows the computation to be more efficient and enables the system of normal equations to be well conditioned. It should be noted that although the GMDH searches for a full combination of candidate terms, there are cases when it will not necessarily provide correct structural model. The reason this might occur is that an output can be accurately described by as few as four ARMA terms, but none of these four terms when paired would produce a small error, as computed according to Eq. 2. Thus, in this case, these four terms would not be selected as the true terms and consequently one would miss the true FIGURE 1. Three-layer iterative process of the GMDH algorithm. Each layer corresponds to the number of iterations.

4 ARMA MODEL PARAMETER ESTIMATION 95 model. A simulation example portraying this scenario is presented in Table 3 of the Simulation Results section. Note that the method outlined is also applicable to subclasses of linear ARMA models such as AR and MA models. SIMULATION RESULTS In this section we demonstrate the effectiveness of the developed algorithm for estimating parameters of the ARMA model. We compare the GMDH results with parameters obtained via the methods of FOS and the least squares. We chose the FOS since, as far as we are aware, it is one of the most accurate algorithms available. The least squares is chosen since it is still one of the most widely used methods in practice. Note that the FOS employs its own robust automatic model order selection 12 whereas the least-squares method relies on the use of model order selection schemes such as the Akaike information criteria AIC. 1 For all simulation examples, we iterate the GMDH algorithm 3 times as discussed in the Methods section. In addition, for all simulation examples to follow, data was sectioned into a training and testing data set with testing data used only in step 3 earlier to measure the mean-square errors MSE value of the model obtained with the training data set. Effects of Incorrect Model Order Selection For the first simulation example, the following linear ARMA model was simulated with Gaussian white noise GWN as the input, x(n), so that the output, y(n), contained 1000 data points y n 0.54y n y n y n x n 0.43x n x n x n 3. The objective is, based on only the measured input signal, x(n) and the output signal, y(n), estimate the parameters of the earlier equation as accurately as possible. Although the true ARMA model order for the earlier process is three output lags y(n 1)...y(n 3) and three input lags x(n),x(n 1),..x(n 3), we purposely selected an incorrect model order of ten output lags y(n 1),,y(n 10) and ten input lags x(n),x(n 1),..,x(n 10) for all three methods. In real life settings, a true model order is unknown a priori, thus, the real test of an algorithm is to examine its efficacy with an incorrect model order. The model orders for the least-squares method were determined according to the AIC and were correctly determined to be three output lags and three input lags. Figure 2 shows the plot of the difference between mean-square error values for 4 FIGURE 2. Bar graph of the difference between MSE values for adjacent nodes as a function of the lower node numbers. adjacent nodes versus the lower node number as a venue for determining model order for the GMDH algorithm. As shown in Fig. 2, we decided to terminate the computation of the GMDH algorithm after six nodes since this corresponded to the minimum error value concomitant with a sufficient number of nodes to reflect inclusion of most of the candidate terms in the model. Adjacent node 2, although it has a small error value, was not chosen since it resulted in only a few model candidate terms and the normalized mean-square errors would have been greater than the choice of adjacent node 6. It should be pointed out that it is not crucial that we select the minimum error value as the slightly higher error values associated with four and five nodes all do result in the same correct ARMA model parameters. The reason we obtain the same model coefficients whether we choose four, five, or six nodes is that, for example, nodes 5 and 6 may contain model terms that are already contained in node 4. Thus, no new model terms from those of node 4 have been added in nodes 5 and 6, resulting in the same coefficients whether node 4 or node 6 is chosen. Comparison of the results based on the three methods is shown in Table 1. Only the new algorithm GMDH and the least squares are able to identify correctly the true parameters in the system model without concomitant estimation of the incorrect model terms. The leastsquares and GMDH model correctly found coefficients of zero for all the spurious model orders, so they are not shown in Table 1. The FOS, however, performs the worst for this simulation as exemplified by the incorrect estimation of parameters and spurious model terms. It should be pointed out that only in certain cases, such as this example, does the FOS not correctly identify the parameters. In many instances, the FOS does provide accurate parameter estimates despite inaccurate model order selections. 4,10,11

5 96 K. H. CHON and S. LU TABLE 1. Comparison of GMDH, FOS, and the least squares with an incorrect model order selection. Model terms y(n 1) y(n 2) y(n 3) y(n 5) x(n) x(n 1) x(n 2) x(n 3) x(n 5) True values GMDH FOS Least squares To further examine the efficacy of the GMDH, consider an ARMA process described by the equation y n 0.546y n y n y n y n x n 0.454x n x n x n x n 4, 5 where the input x(n) is the same input as used in Eq. 4. For all three methods compared, we chose an incorrect model order of AR 10 and MA 10. Table 2 shows the results. We observe that only the least-squares based on the AIC model order selection was not able to identify the correct parameters since the AIC resulted in model order of AR 3, MA 7. The GMDH provided the correct model terms except for the term y(n 4), which in turn resulted in incorrect parameter estimates. The FOS did not fare as well as the GMDH method. It not only missed some of the model terms but it incorrectly identified terms that are not in the true model. The normalized mean-square errors NMSE values for the GMDH and FOS are and , respectively. As shown in Table 2, the GMDH performed the best compared to the other two methods. This example is provided to show that the GMDH does not always provide accurate parameters, especially for a simple example such as this. The reason the GMDH does not necessarily provide correct parameter estimation in this example is quite simple. In Table 2, the term y(n 4) when paired with other candidate terms, produced a small reduction in error, thus was rejected and missed the true model. Thus, the GMDH is not to be considered a global optimal search method; an optimal search method would require testing for the minimum error across all possible subsets of candidate functions within the candidate function space. ARMA Model with Additive Noise and Incorrect Model Order Selection To examine the effectiveness of the GMDH in the case of noise contamination, the output signal of Eq. 4 was corrupted with two levels of additive GWN, resulting in signal-to-noise-ratios SNR of 10 and 0 db. The SNR was determined by the ratio of the variance of the signal divided by the variance of the noise signal. The model order was incorrectly selected to be ten output and ten input lags for all methods compared. This is a challenging and realistic scenario; not only is the model order purposely incorrectly chosen, but the output signal is heavily corrupted by noise. The results of additive noise for the GMDH, FOS, and the least squares are shown in Table 3. With the SNR of 10 db, the GMDH correctly provides parameters that are associated with only the true model terms but this is not the case for the FOS; the FOS missed a model term, y(n 3), and incorrectly picked an additional term, x(n 4). The leastsquares approach based on the AIC resulted in a model order selection of AR 7 and MA 7. Since the leastsquares method does not have automatic model order selection, it resulted in estimates of terms and coefficients not included in the actual system in Eq. 4 due to an overdetermined model order assumption. The NMSE for all methods are comparable, however, it should be noted that for the least-squares, the comparable NMSE was obtained with 15 parameters whereas for both the GMDH and FOS, the NMSE values were obtained with only 7 parameters. More qualitative differences between the methods compared emerge when the SNR is 0 db. The GMDH again provided only the correct model terms as was the case when the SNR was 10 db. The FOS missed three model terms, y(n 1), y(n 2), and y(n 3). In addition, the FOS incorrectly picked three additional model terms, y(n 5), x(n 4), and x(n 5). The AIC model order selection resulted in AR 0 and TABLE 2. Comparison of GMDH, FOS, and the least squares with an incorrect model order selection. Model terms y(n 1) y(n 2) y(n 3) y(n 4) y(n 6) y(n 7) x(n) x(n 1) x(n 2) x(n 3) x(n 4) x(n 6) x(n 7) True values GMDH FOS Least squares

6 ARMA MODEL PARAMETER ESTIMATION 97 TABLE 3. Comparison of GMDH and FOS with additive noise and an incorrect model order selection. Model terms y(n 1) y(n 2) y(n 3) y(n 5) x(n) x(n 1) x(n 2) x(n 3) x(n 4) x(n 5) True values GMDH (10 db) FOS (10 db) Least squares (10 db) GMDH (0 db) FOS (0 db) Least squares (0 db) MA 7, and the resulting coefficients via the leastsquares are provided in Table 3. With significant noise added, the least-squares based on the AIC completely missed the AR terms and then compensated by including more MA terms than necessary. Because the FOS was designed to reduce the NMSE, despite incorrect model terms, the NMSE value obtained for the FOS is comparable to the GMDH and the least squares. ARMA Model with Missing Terms The next example considers an ARMA process with missing terms and 1000 data points generated for the output y(n): y n 0.25y n 1 0.1y n 2 0.1y n 6 0.8x n 0.3x n x n 6. Although the true ARMA model order for the process of Eq. 6 is AR 6 and MA 6, we have again purposely selected an incorrect model order of ARMA 10,10 for all three methods. The Akaike information criteria correctly identified model order of ARMA 6,6. The FOS, due to its automatic model order search criterion, also identified only the correct parameters. The GMDH method provided correct parameter estimates for 4 9 nodes. Thus, similar to the previous example, it is not imperative that the exact minimum error value is used as a criterion in deciding when to terminate the GMDH algorithm. Thus, despite an incorrect model order assumption, all three methods provided only the correct parameters. This is clearly an impressive result by all of the methods compared considering the fact that a grossly incorrect model order was chosen a priori. 6 ARMA Model with Dynamic Noise and Incorrect Model Order Selection The second simulation discussed included additive noise. The noise source was added after the output sequence, y(n), had been generated. However, noise sources can also be of a dynamic nature. Dynamic noise refers to a feedback process where the noise source is recursively added to the output so that the current and future output values are dependent on the past states of the input and noise signals. Thus, to examine the effect of dynamic noise consider an ARMA process described by the difference equation y n 0.54y n y n y n x n 0.43x n x n x n e n 0.3e n 1. Both the input x(n) and the noise component terms e(n) were generated using 1000 Gaussian white noise data points that were independent from each other. The noise component e(n) was selected to give a SNR of 10.4 db. We chose an incorrect model order of AR 10 and MA 10 for all three methods compared. Table 4 shows the results of the coefficients estimated via the three methods. It is clear that only the GMDH provided all of the model terms correctly. The least-squares method completely ignored the AR terms and included a few extra MA model terms that are not in the true model, owing to the AIC model order selection of AR 0, MA 7. The FOS missed the y(n 3) term and provided two extra terms, x(n 4) and x(n 5). The NMSE values are about 9% for all three methods. 7 TABLE 4. Comparison of GMDH and FOS with dynamic noise and an incorrect model order selection. Model terms y(n 1) y(n 2) y(n 3) x(n) x(n 1) x(n 2) x(n 3) x(n 4) x(n 5) x(n 6) x(n 7) True values GMDH FOS Least squares

7 98 K. H. CHON and S. LU CONCLUSIONS We presented a new algorithm, the group method of data handling, for estimating the parameters of ARMA models. The algorithm can be extended to estimate subclasses of linear ARMA models such as AR MA models, as well as nonlinear ARMA models. In addition, the algorithm can be extended to model the prediction error terms which will significantly improve the model prediction. Various simulation examples have shown the efficacy of the GMDH method. In certain cases of noiseless systems, the least squares based on the AIC model order selection is superior to FOS. However, when the system output is confronted with noise either additive or dynamic, the GMDH method performed best followed by the FOS and the least squares in providing only the true model terms. If one is interested in obtaining the smallest mean-square error with the fastest computation time, then simulation examples suggest that the FOS would be the ideal method to be use. However, if obtaining the most accurate representation of the true model is the necessity, then the GMDH method should be the algorithm of choice, at least for the simulation examples considered above so far. As has been reported in the literature, the least-squares method based on the AIC model selection can work well if the system is noise free. However, the accuracy of the least-squares approach to ARMA model parameter estimation degrades considerably when the system output is corrupted with any type of noise source. For the simulation examples considered, the GMDH, unlike the other two methods compared, provides accurate model terms even in cases with significant noise perturbation as well as incorrect model order selection. ACKNOWLEDGMENT This work was supported by the Whitaker Foundation. REFERENCES 1 Akaike, H. Power spectrum estimation through autoregressive model fitting. Ann. Instrum. Stat. Math. 21: , Baker, J. E. Adaptive selection methods for genetic algorithms. In: Proc. First International Conf. Genetic Algorithms and Their Application, edited by J. J. Grefenstette, Hillsdale, NJ: Erlbaum, Chon, K. H., and R. J. Cohen. Linear and nonlinear ARMA model parameter estimation using artificial neural networks. IEEE Trans. Biomed. Eng. 44: , Chon, K. H., M. J. Korenberg, and N. H. Holstein-Rathlou. Application of fast orthogonal search to linear and nonlinear stochastic systems. Ann. Biomed. Eng. 25: , Chon, K. H., R. J. Cohen, and N. H. Holstein-Rathlou. Compact and accurate linear and nonlinear autoregressive moving average model parameter estimation using Laguerre functions. Ann. Biomed. Eng. 25: , Farlow, S. J. The GMDH algorithm of Ivakhnenko. Am. Stat. 35: , Ivakhnenko, A. G. Polynomial theory of complex systems. IEEE Trans. Syst. Man Cybern. 1: , Ivakhnenko, A. G. Group method of data handling A revival of the method of stochastic approximation. Sov. Automatic Control 13:43 71, Ivakhnenko, A. G., G. A. Ivakhnenko, and J. A. Muller. Self-organization of neuronets with active neurons. Pattern Recognition and Image Analysis 4: , Korenberg, M. J. Fast orthogonal identification of nonlinear difference equation and functional expansion models. Proc. Midwest Symp. Circuit Sys. 1: , Korenberg, M. J. A robust orthogonal algorithm for system identification and time series analysis. Biol. Cybern. 60: , Korenberg, M. J. Fast orthogonal algorithms for nonlinear system identification and time-series analysis. In: Advanced Methods of Physiological System Modeling, edited by V. Z. Marmarelis, Los Angeles: Biomedical Simulation Resource, 1989, Vol. 2, pp Mitchell, M. An Introduction to Genetic Algorithms. Cambridge, MA: MIT Press, 1998.