Chaotic Time Series Prediction by Fusing Local Methods

Transcription

1 Chaotic Time Series Prediction by Fusing Local Methods Abstract In this paper, a novel algorithm is proposed to predict chaotic time series. The chaotic time series can be embedded into state space by Takens embedding theorem. The one dimensional data is mapped to a higher dimensional space that provides precise information about the chaotic time series. The upsampling algorithm is used to find more precise nearest neighboring points. Two algorithms which provide accurate results without the knowledge of the underlying dynamics and fuzzy fusion algorithm are employed for one-step and multi-steps ahead forecasting. Simulation results from three typical chaotic time series demonstrate that our method is effective for chaotic time series. Keywords-chaotic time series; fuzzy fusion; embedding theorem; upsampling algorithm; multi-step ahead ; I. INTRODUCTION (HEADING 1) Time series widely exist in our natural world, including macroeconomics, finance, traffic flow, crowd flow, water flow and so on. There are many works focus on modeling time series by linear models, such as auto regressive moving average (ARMA), auto regressive integrated moving average (ARIMA). Strictly speaking, most of the time series can not be accurately depicted by linear models, since the nonlinearity is ubiquitous in the dynamic model. There exists a rich literature for time series. These mothods are classified into two types, linear and nonlinear methods, based on the type of functions that are used in. In linear method, The ARMA method involves two parts, Auto Regression and the Moving Average. It takes into consideration the regression models of data and also the moving average for analyzing the time series data. The ARIMA method is obtained by integrating the ARMA model. Nonlinear method includes radial basis functions, neural networks, and polynomials [1]. The nonlinear methods mainly deal with chaotic time series which is more complex. Chaos theory emerged in 1960s since Lorenz revealed the butterfly effect in studying the weather. After that, chaos have been widely studied and lots of important concepts has been introduced, such as the embedding dimension, embedding delay, chaotic invariants and reconstruction. Application of chaos theory can be found in the area of Yong Wang, Shiqiang Hu* School of Aeronautics and Astronautics Shanghai Jiao Tong University, Shanghai wysjtu2008@gmail.com, sqhu@sjtu.edu.cn production control, human action recognition, signal processing, economic planning and many other fields. The most striking feature of chaotic system is the unpredictability of its feature which is called the sensitivity dependence on initial conditions or butterfly effect. From mining the past to predict the future, the chaotic time series has been widely used in weather forecasting, electric power etc. A number of algorithms have been proposed so far with their own merits and limitations. These methods can be divided into two categories, one-step ahead and multi-step ahead. The later one which is more difficult has to deal with accumulation of errors and uncertainty. For one-step ahead, [6] combined different predictors including the multi-layer perceptron neural network, radial basis function (RBF) neural network, fuzzy inference system, recurrent neural network, Volterra filter, and local linear predictor to predict a chaotic time series. And the linearly constrained least square fusion method is employed to improve the performance of. The multiresolution learning is used in training of predictors. That is signal is decomposed and approximated on different levels of details. And the training process learned from coarse to finest. For multi-step ahead, [8] adopted recursive strategy, which firstly obtained a one-step ahead model and then the estimation value is fed back as an input for the next forecasting. In [9] the authors used a direct strategy that estimated a set of models, each returned a value. There are works that combined the two previous strategies which used different models at each step and the s from previous steps are introduced into the input set. [10] presented an approach for short term for biological and physical phenomenon without knowing the characteristic of the dynamical system. [11] have pointed out that direct strategy is questionable as mapping a p steps into the future would usually be more complicated than recursive strategy. In [2] the authors have shown that recursive strategy performed better on short term. Motivated by the works mentioned above, we use two local algorithms and recursive strategy to forecast chaotic time series. Compared with [6], we make four differences. Firstly, we use two local algorithms. The chaotic * corresponding author address: sqhu@sjtu.edu.cn This paper is jointly supported by the National Natural Science Foundation of China

2 system produces very irregular data which is very different from linear system and weakly nonlinear system. It is not an easy task to build such a model due to the high volatility of the underlying laws behind the chaotic time series. We infer data from nearest neighboring points which is similar to data driven method, hence avoid approximating the function of the dynamical system. Secondly, instead of multiresolution learning, upsampling is employed to interpolate the time series. The higher the upsampling rate, the more detail information of the time series we can obtain, which is especially helpful when the systems fluctuate rapidly. Thirdly, we use fuzzy fusion rule which fuse the two data smoothly. Last but not least, we do not only predict the data one-step ahead, but also solve multi-step ahead. The paper is organized as follows. Section 2 gives a brief background material for state space reconstruction and the method for finding nearest neighboring points. Two algorithms and a fuzzy fusion algorithm are presented in section3. Section 4 gives the experimental results and performance analysis. Finally, conclusions are drawn in section 5. II. BASIC CONCEPTS First let us introduce a few concepts used in our wok. A. state space reconstruction [3] In the time domain, chaotic time series display stochastic behavior. However, it is characterized by the values of a deterministic process in the embedding state space. Given a time series,,,, the state space can be reconstructed according to Takens theorem [7]. Specifically, the time series can be mapped to a higher dimensional state space and hold almost the same structure with the original time series if we select the parameters embedding time delay and embedding dimension d appropriately. Once the two parameters are determined, the state can be written into a matrix. (1) B. Finding nearest neighboring points To find the nearest neighboring points of the state in the reconstructed state space. We first compute the state and its 1 prior state 1,2,, 1 with a predefined metric.,i 1,2,,n1 (2) All the distances are arrayed in an increasing order. And then find the nearest neighboring points. C. Upsampling Finding nearest neighboring points is a critical step for chaotic time series as we need to use the trajectory of neighbors to predict chaotic time series. If the time series is coarsely represented by discrete points, the neighboring point can only be found with coarsely precision. In order to increase the accuracy, interpolation is employed to find the nearest neighboring points with greater precision. The experiments demonstrate that the will be more accurate if the system is interpolated by such way. III. OUR PREDICTION METHODS According to [11] and [12], they have shown that methods based on local mode which presented a number of favorable properties produced more accurate results than other methods. Our local method is based on assumption that equal states have an equal future, and similar state will evolve similarly, at least over short times according to the chaos theory. A. kernel regression algorithm The main idea is to predict the point by using a weighted average of dynamics of neighboring points in the state space. The weights of the neighboring points are defined as: _ (3) _ where _ is the minimum of the neighboring distance, is the number of neighboring points, is predefined parameter, is the ith nearest neighboring point distance. Hence, the next predicted point is given by, (4) x where is one of the K nearest neighboring points of. is the number of neighboring points of. B. Largest Lyapunov Exponent (LLE) algorithm Lyapunov exponent is a dynamical invariant of the attractor, and measures the exponential divergence of the nearby trajectories in the phase space. The LLE depicts the distance of two neighboring points after steps. The distance d between points and its neighboring point is dt xi xt. After steps evolved, the distance between the two points is d t xi pxtp. The relation between and is de, where is LLE. C. Fuzzy fusion rule [13] Fuzzy rule is defined as follows: : if is and is that is, 1,2,, where is the j th fuzzy rule, is the number of fuzzy rule,, and are the input and output of the fuzzy logic respectively., and are fuzzy linguistic term characterized by membership function. In the algorithm, the input of the fuzzy logic is the error between the and the real value. For one-step ahead, errors and in current step between the value and the real value are normalized as e, 708

3 e. The output is the weight in the next step. According to the fuzzy logic rule, data with large error is given smaller weight while data with small error is given larger weight. For multi-step ahead, we use a similar approach. In order to determine the weight of the two algorithms, the real value in the chaotic time series is used to compare with the estimated data. The errors are used in the same way as in one-step ahead to determine the weight of the two results in the next step. D. error analysis In this paper, the relative mean square error (RMSE) is used to evaluate the proposed method. (5) where is th chaotic time series value, is th chaotic time series point real value. is the length of predicted chaotic time series. IV. EXPERIMENTAL RESULTS AND DISCUSSION The experimental results on three different chaotic time series data are presented. A. Ikeda chaotic time series Ikeda chaotic time series is generated by the following equation: x 1cossin (6) y sin t cos (7) where µ 0.9, x 0.1, y 0.1 is the initial value for x and y respectively. The initialization condition is 0.46/1 (8) (9) (10) Integrate the equation (9) and (10) to obtain the following equation, xi11cos 1sin (11) 1 1 (12) In the one-step ahead experiment, we use 400 data for training and 100 for testing, respectively. The errors of the three algorithms are 0.16, and The results are shown in figure 1. The first row is the original data. The second row is the kernel regression algorithm result and errors. The third row is the LLE algorithm result and errors. The fourth row is the fuzzy fusion algorithm result and errors. In the one-step ahead with experiment, we use 200 data for training and 50 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 2. Compared with figure 1 and figure 2, it shows the upsampling algorithm improves the results. Figure 1: Ikeda chaotic time series one-step ahead and errors Figure 2: Ikeda chaotic time series one-step ahead and errors with. In the multi-step ahead experiment, we use 400 data for training and 50 for testing. The errors of the three algorithms are 0.2, 0.3 and 0.22 respectively. The results are shown in figure 3. Figure 3: Ikeda chaotic time series multi-step ahead and errors In the multi-step ahead with upsampling rate 5 experiment, we use 50 data for training and 10 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 4. In the multi-step ahead with upsampling rate 20 experiment, we use 50 data for training and 10 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure

4 Figure 4: Ikeda chaotic time series multi-step ahead and errors with upsampling rate 5. Figure 6: Mackey-Glass chaotic time series and errors In the one-step ahead with upsampling rate 5 experiment, we use 200 data for training and 50 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 7. Figure 5: Ikeda chaotic time series multi-step ahead and errors with upsampling rate 20. Compared with figure 3 and figure 4, we find the improvement of upsampling algorithm is not obvious. However, the performance is greatly promoted in figure5. The RMSE of the three predictors is shown in Table 1. Table 1 Ikeda chaotic time series errors Predictor kernel regression LLE Errors One-step ahead One-step ahead with Multi-step ahead Multi-step ahead with upsampling rate Multi-step ahead with Fusion B. Mackey-Glass chaotic time series The Mackey-Glass chaotic time series is described by the following: x /1 (13) where 0.2, 0.1, 17, In the one-step ahead experiment, we use 400 data for training and 100 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 6. Figure 7: Mackey-Glass chaotic time series one-step ahead and errors with upsampling 5. In the multi-step ahead experiment, we use 600 data for training and 50 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 8. Figure 8: Mackey-Glass chaotic time series multi-step ahead and errors In the multi-step ahead with experiment, we use 400 data for training and 10 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 9. The RMSE of the three predictors is shown in Table

5 Figure 9: Mackey-Glass chaotic time series multi-step ahead and errors with. Table 2 Mackey-Glass chaotic time series errors Predictor kernel regression LLE Fusion Errors One-step ahead One-step ahead with upsampling rate 5 Multi-step ahead Multi-step ahead with C. Lorenz chaotic time series The Lorenz chaotic time series use in this simulation is generated by the following equation: x (14) y (15) z (16) where 16, 45.92, 4, y 1, 0, 1, time step In the one-step ahead experiment, we use 400 data for training and 100 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 10. Figure 11: Lorenz chaotic time series one-step ahead and errors with upsampling 5. In the multi-step ahead experiment, we use 600 data for training and 50 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 12. Figure 12: Lorenz chaotic time series multi-step ahead and errors In the multi-step ahead with experiment, we use 400 data for training and 50 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 13. The RMSE of the three predictors is shown in Table 3. Figure 10: Lorenz chaotic time series and errors In the one-step ahead with upsampling rate 5 experiment, we use 200 data for training and 100 for testing. The errors of the three algorithms are , and respectively. The results are shown in figure 11. Figure 13: Lorenz chaotic time series multi-step ahead and errors with. Table 3 Lorenz chaotic time series errors Predictor kernel regression LLE Fusion Errors One-step ahead

6 One-step ahead with upsampling rate 5 Multi-step ahead Multi-step ahead with D. Discussions: Let us briefly summarize the findings of experiments. (a) In the three experiments, the kernel regression results are generally more accurate than LLE results. One of the reasons is the values of LLE are not precisely determined. There are several methods to estimate the LLE. And each results of the method are different. Another reason is the LLE algorithm is an approximating algorithm. (b) The upsampling step is effective, especially for multi-step ahead. In one-step ahead, the error is small and will be corrected by the real data. In multi-step ahead, no real data for reference. And one feature of the chaotic time series is the butterfly effect mentioned in the first section. Therefore, a small error will be magnified quickly in the multi-step ahead. (c) The experiments show that the bigger the upsampling, the more accurate of the results. Meanwhile, the computation burden will increase. There is no much relation between results and the training data only when the length of training data is a certain large. In the multistep ahead the upsampling algorithm promote the performance significantly. In the signal processing, the Nyquiest theorem [5] points out to reconstruct a signal the sampling frequency should be at least two times higher than the frequency of the original signal. Correspondently, there should be a relationship between the upsampling frequency and the frequency of the chaotic time series. When the upsampling frequency compared with the frequency of the chaotic time series higher than a certain value, the precise nearest neighboring points can be found. The frequency of the Ikeda data is the highest while the Lorenz is the lowest. In the experiments with upsampling, the Ikeda data need higher upsampling frequency than the other two data to maintain the error small. While the of Lorenz data which with lowest frequency among the three data achieve good performance even with low upsampling frequency. (d) In several experiments the results of kernel regression is more accurate than the LLE result and even fusion result. We can observe that the errors of kernel regression generally are much smaller than the errors of LLE. As the fuzzy fusion include the weight of LLE, the performance of fusion algorithm degenerate. It shows that our fusion algorithm may not be the optimal. When we do data fusion, delete the data of large error is better than do directly. This indicates us to find a better criterion to judge the credibility of the fusion data which is out of the scope of this paper. (e) From the one-step ahead, the fusion algorithm can improve the result while in the multi-step ahead. The fusion is also failed as the errors of kernel regression and LLE are large. V. CONCLUSIONS We have presented kernel regression algorithm and LLE algorithm for chaotic time series in this paper. A fusion based on fuzzy rule is also provided. Three chaotic time series are used to verify our algorithm and promising results are achieved. We also compared the upsampling chaotic time series with original data. The results confirm our conjunction that the upsampling algorithm promotes the results. Future works includes finding out proper upsampling rate for multi-step ahead and multivariate chaotic time series which is a more challenging research field. REFERENCES [1] H. Kantz and T. Schreiber, Nonlinear Time Series Analysis, Cambridge University press, [2] Casdagli M. Nonlinear of chaotic time series. Physica D 1989;35: [3] Abarbanel Henry DI. Analysis of observed chaotic data. New York: Springer-Verlag; [4] Abarbanel Henry DI, Brown R, Kadtke JB. Prediction in chaotic nonlinear systems: methods for time series with broadband Fourier spectra. Phys Rev A 1990;41: [5] Monson H. Hayes. Statistical Digital Signal Processing and Modeling. John Wiley & Sons, Inc., [6] M.Aly and H.Leung, Chaotic time series using data fusion, International Conference on Data Fusion (FUSION2001), Montreal, Canada, August [7] Takens F. Detecting strange attractors in turbulence. In: Rand DA, Young LS, editors. Dynamic systems and turbulence, Warwick, Lecture notes in mathematics. Berlin: Springer Verlag; p [8] Haibin Cheng, Pang-Ning Tan, Jing Gao, and Jerry Scripps. Multistepahead time series. In Wee Keong Ng, Masaru Kitsuregawa, Jianzhong Li, and Kuiyu Chang, editors, PAKDD, volume 3918 of Lecture Notes in Computer Science, pages 765{774. Springer, [9] A. Sorjamaa, J. Hao, N. Reyhani, Y. Ji, and A. Lendasse. Methodology for long-term of time series. Neurocomputing, 70(16-18):2861{2869, October [10] G. Sugihara, R.M. May,Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series, Nature 334 (1990) [11] J. Doyne Farmer and John J. Sidorowich. Predicting chaotic time series. Physical Review Letters, 59(8):845{848, August [12] Tim Sauer. Time series by using delay coordinate embedding. In Andreas S. Weigend and Neil A. Gershenfeld, editors, Time Series Prediction, Santa Fe Institue Studies in the Sciences of Complexity, pages Addison-Wesley, [13] T. Takagi and M. Sugeno. Fuzzy identification of systems and its applications to modeling and control. IEEE Transactions on Systems, Man, and Cybernetics, 15(1):116{132,