User Activity Recognition Based on Kalman Filtering Approach

Transcription

1 User Activity Recognition Based on Kalman Filtering Approach EEC 592, Prosthesis Design and Control Final Project Report Gholamreza Khademi December 2, 214 Abstract Different control methods are used to control a prosthetic leg. Finite-state machine is one of the well-known strategies applied in the control design. Since, the dynamic and kinematics of human motion is changing during various walking modes, a separate controller is used for different phases. Finite-state machine is used for this purpose. To let the prosthetic leg choose the suitable control mode and transit from one phase to another, the user intent recognition is required. The concept behind user intent recognition is to compare the current state of prosthesis to the trained probabilistic models obtained for each activity such as standing, walking, sitting and etc. to identify the activity a user is performing. The main focus of this project is to use the mechanical information of sensors such as joint position, velocity and ground reaction force (GRF) to train a model to determine the user activity recognition. It is assumed the human walking is quasiperiodic, and a Fourier series based model is fitted to each activity mode. The coefficients of Fourier series are different for each activity. The difference is a rich feature to classify various walking modes. Two different approaches are implemented to estimate the coefficients of Fourier series. First, offline user activity recognition is developed by solving a minimization problem. Then, a kalman filtering approach is implemented to estimate the coefficients of Fourier series model. The model is trained for standing, level walking, ramp up and ramp down with slope of 5 degree. Some human walking test data is applied to the system; and performance of obtained model is verified by identifying the user s activity. 1. Introduction According to the literature, most of user intent recognition is based on Electromyography (EMG) signals. In [1, 2], EMG based user intent recognition is applied and signals from the stump residual is collected to determine the user purpose. This is done for level walking, ramp up and down with various slopes. However, in some papers, mechanical information from sensors on the prosthetic leg is used to determine the user intent. In [3], an EMG based pattern recognition is tested on different able-bodied and above knee amputation subjects. They could obtain 9% accuracy to identify the user intent. They improved the accuracy by fusion of mechanical information from sensors and the EMG signals. Moreover, in [4] sensors information for a patient with transfemoral amputation is recorded for different locomotion modes and showed 1% accuracy in detecting the transition. The application of user intent recognition is not merely restrained to human prosthetic limb. In [5], the measured data from the cellphone accelerometer is utilized to perform activity recognition. Either EMG or sensor based user intent recognition are similar to each other. Actually, both strategies attempt to obtain specific features from the collected data to train a model and classify different locomotion modes. Then, the resulting model is utilized to conclude the user s intent. General architecture of user intent recognition system is shown in Figure 1. Page 1 of 15

2 Figure 1: General architecture of user intent recognition system, taken from [3]. In the present project, the same steps as shown in general architecture of user intent recognition system are followed. An appropriate set of sensors data will be selected as the input to the user intent recognition system. The selected input sets are knee angle and hip angle for the standing, level walking, and ramp up and down with the slope of 5 degree. The data are provided by VA. Then, the model is trained based on these sets of data for each activity. Since the human walking is similar to a periodic signal, it is possible to fit a Fourier series to each periodic signal. Thus, the Fourier series is used as the model. The coefficients of Fourier series are different at each walking mode. Two approaches are used to obtain the coefficients. First, the Fourier series is fitted to knee and hip angle for different walking modes. This is done by solving a minimization problem. Levenberg-Marquardt algorithm is used for this purpose. Then, the coefficients are labeled and classified with a simple classification method. This approach is called offline user activity recognition, since it requires the whole human gait data at a time to fit a Fourier series. This is the main drawback of this method. To resolve the problem, kalman filtering approach is proposed to obtain the coefficients at each time step. In this way, it is possible to have real time user intent recognition. The project is outlined as follows. In section 2, offline user activity recognition is introduced. Moreover, Levenberg-Marquardt Algorithm and k-nn classification method is introduced in this section. Online user activity recognition and kalman filter is described in section 3. Finally, conclusions and future work are discussed in section 4.

3 2. Offline user activity recognition Human gait data is similar to a periodic signal. It is possible to fit a Fourier series to each periodic signal. Figure 2 shows a normal able-bodied gait data for several strides. It is obvious the data trend is periodic. Figure 2: normal able-bodied gait data for several strides, taken from [6]. Fourier series is used as the model. The coefficients of Fourier series are different at each walking mode. The Fourier series is fitted into the knee and hip angle for different walking modes. This is done by solving a minimization problem. The Levenberg-Marquardt algorithm is used for this purpose. Then, the coefficients are labeled and classified with a simple classification method. This approach is called offline user activity recognition, since it requires the whole human gait data at a time to fit a Fourier series. This is the main drawback of this method. After training step, the model is used to identify the user s activity for a sample test data. The offline user activity recognition approach is summarized in Figure 3. Figure 3: offline user activity recognition approach In the following subsection, 2.1, the general form of Fourier series is reviewed. Then, in subsection 2.2, Levenberg-Marquardt algorithm is applied to fit Fourier series into the knee and hip angle. k-nn classification method is introduced in subsection 2.3.

4 2.1. Fourier series A periodic function,, can be represented as a Fourier series with an unlimited sum of cosine and sine terms. The study of Fourier series is a part of harmonic analysis. Fourier series is a useful tool to approximate any periodic function with a preferred or practical accuracy. In Figure 4, a square wave is approximated by sum of cosine and sine. Figure 4: a square wave approximated by Fourier series The Fourier series of a given periodic function, with period is 1 2 cos sin Levenberg-Marquardt algorithm Levenberg Marquardt method (LM) was introduced by Levenberg and Marquardt, suggests a solution to the minimization problem of a nonlinear function [7]. The LM algorithm is an iterative gradient based method that finds the local optimum in a search domain. The LM algorithm combines the simple steepest descent and the Gauss Newton algorithms. It has the fast convergence benefit of Gauss Newton method and the stability of steepest descent method. The LM method is wildly used to solve nonlinear least square optimization problems. Curve fitting problems can be defined in terms of nonlinear least square optimization problem. The LM optimization method finds the local optimum value for the decision variables, so as to minimize any objective function. The curve fitting minimization problem can be defined as follows [8]: minimize 1 2 Where is given as the objective function and represents decision variable vector which is Fourier series coefficients. is given as the error vector. The error is defined as equation (3), where is the human gait data like knee and hip angle and is the Fourier series model. 3 The gradient based algorithms require the derivative of objective function with respect to the decision variables. The derivative of objective function is given by 2

5 is called Jacobian matrix. The Jacobian matrix is given as Where is the number of weights and is the number of decision variables. The update rule of LM approach could be written as is known as the second order Hessian matrix and the term is added to make sure the Hessian matrix is always invertible. is an identity matrix and is the damping parameter. Selection of damping parameter is important in performance and fast convergence of LM algorithm. If a step is acceptable, it means there is decrease in objective function. Thus, it is preferred to decrease damping parameter to have more speed in convergence toward the correct direction. Similarly, if the step is unaccepted, damping parameter must be increased until an acceptable step is achieved. The LM method pseudocode is summarized as follows. Initialize,,, and While not (termination criterion) Calculate the objective function: Calculate the derivative of objective function: Calculate the Hessian matrix: Loop to adjust Update decision variable with LM rule: Calculate objective at new obtained point: If < then is acceptable & & Next iteration If > then is unacceptable & & Loop to adjust Next iteration 1 The curve fitting problem is solved by LM optimization method. A Fourier series with 2 frequencies is fitted into the knee angle. The optimum values of coefficients are obtained. The coefficients are compared with their initial values in Table 1. The LM method can stuck in the local optimum. To prevent that LM method is ran for different random initial guess. Table 1: coefficients of Fourier series with 2 frequencies fitted into the knee angle Simulation Case RMSE LM method Initial , 1 Optimum In Figure 5, the Fourier series with optimum coefficients is plotted versus the knee angle. Moreover, the variations of coefficients for achieving their optimum values are shown in this Figure

6 Coefficients a a1 b1 a2 b2 w Knee Angle Knee Angle Fitted Knee Angle Iterations Time (Sec) Figure 5: fitted Fourier series versus knee angle (Right), coefficients update (Left) An important question about this method will remain unanswered. How many frequencies are needed for the Fourier series to achieve a suitable curve fitting? To reply this question, the usual system order selection approach will be used. There are different ways to select an appropriate order for the estimated models. There are two main principles: goodness of fit and parsimony. According to the goodness of fit principle, appropriate order is the one that fits the model output to the reference output as close as possible. The parsimony principle selects the simplest model with reasonable fitness. There are various mathematical criteria allow to select the best model order. These methods create balance between the model fit and model simplicity. The error of curve fitting is computed for different number of frequencies. Figure 6 shows the root mean square error (RMSE) with respect to number of Fourier series frequencies RMSE No. of Frequency Figure 6: RMS error with respect to number of frequencies

7 According to the principle of Parsimony, the Fourier series with 3 frequencies is used. By this choice, it is tried to make a balance between the models fit and model simplicity. The Fourier series with different frequencies is fitted to the knee angle in Figure 7. In this Figure, it is shown that the fitness of Fourier series with 3 frequencies is approximately close to the one with 1 frequencies. To show that numerically, LM method is applied to fit the Fourier series. RMS error is computed for different frequencies and placed in Table 2. The RMS error obtained from LM method is compared to the RMS error provided by curve fitting tool in MATLAB. It could be shown that the LM method performs better than MATLAB curve fitting toolbox. It should be noted that MATLAB curve fitting tool could not fit a Fourier series with more than 8 frequencies. Knee Angle y 1 Freq Freq. 3 Freq. 4 Freq Freq. 6 Freq. 7 Freq Freq Freq. 1 Freq. 8 y 6 3 Freq y 1 Freq Time (Sec) Figure 7: Fourier series with different frequencies versus the knee angle Table 2: Fourier Series RMS error with respect to number of frequencies RMSE LM Method MATLAB Cftool 1 Freq Freq Freq Freq Freq Freq Freq Freq Freq Freq As a result, a Fourier series with 3 frequencies is selected to fit into the knee and hip angle for standing, level walking, ramp up and ramp down modes. The Fourier series coefficients are obtained for each mode by solving the LM optimization method. For simplicity, the coefficients which are explicitly different in each walking mode are inserted in Table 3. These coefficients represent the user activity mode. They are used as the features and classified by k-nn classification algorithm. cos sin cos2 cos2 cos3 cos3 7

8 Table 4: Fourier series coefficients for different walking modes Training Phase Knee Angle Hip Angle Standing Level Walking Ramp Up Ramp Down k-nn classification method k-nearest Neighbors algorithm (k-nn) is the simplest machine learning algorithm for classifying a test object. The classification method is divided into training and testing categories. In the training case, the training objects are stored with a class label. In testing phase, the test object is classified based on a similarity measure. In other words, the distance of test object to every training object is calculated. Then, the test object is classified based on the majority vote of its nearest neighbors. If 1, the class of test object is assigned to the nearest training object class. A usual similarity measurement is the distance between the test and training object. The common continues distance function used in the k-nn algorithm is Euclidean distance and Sum of absolute differences. Euclidean Distance Absolute Difference Where is the training vector and is the test vector. An example of k-nn algorithm is shown in Figure 8. There are two classes. The first class is blue squares and the red triangles are the second class. The test example is green circle. For 3, the three nearest neighbors of test example is covered by a solid line circle. It is obvious that the test object is assigned to the triangle class. However, it is assigned to the square class according to the majority vote of its 5 nearest neighbors for 5 [9, 1, 11]. 8 9 Figure 8: If 3, green circle is classified as triangle class. If 5, the circle is assigned to the square class.

9 The Fourier series coefficients obtained by LM method represent the user activity mode. They are used as features and classified by k-nn classification algorithm. The coefficients for 4 standing, 4 level walking, 9 ramp up and 1 ramp down trials are obtained and used as training set. A standing, level walking, ramp up, ramp down, level walking and standing mode are used as test data, respectively. The coefficients are obtained for these test dataset and classified with k-nn. k-nn assigned the correct class to each walking mode. The result is shown in Figure 9. Figure 9: Knee and hip angle for a test walking trial It could be concluded that the knee angle is not suitable sensor information for user activity recognition. Since, the knee angle is quite the same in different activity modes such as level walking, ramp up and down. 3. Online user activity recognition As discussed, it is impossible to apply offline user activity recognition in practice. Since, it requires the whole human gait data at a time to fit a Fourier series. This is the main drawback of this method. To resolve the problem, kalman filtering approach is proposed to obtain the coefficients at each time step. In this way, it is possible to have real time user intent recognition. The online user activity recognition approach is summarized in Figure 1. Finally, k-nn classification method is used to classify the obtained coefficient for various activities. Figure 1: online user activity recognition approach To utilize kalman filter approach for estimation of Fourier series coefficients, a state-space realization is required for the Fourier series. In subsection 3.1, a state-space equation is developed for the Fourier series. Then, the continues-time extended kalman filter is used to estimate the coefficients for different walking mode. These coefficients are used as training dataset.

10 3.1. Fourier series state-space realization A periodic signal can be represented by a Fourier series. Suppose the signal period is. Thus, the angular frequency is given as 2. cos sin It is possible to obtain a linear state-space realization for the Fourier series [12]. R R Where, cos sin sin cos cos2 sin 2 sin2 cos 2 cos sin sin cos Since, the angular frequency is varying for different walking mode; it may be a useful feature for classification. Thus, it is desired to estimate the angular frequency, too. As a result, the statespace is changed from linear into nonlinear. A Fourier series with 3 frequencies is again used to fit to the human gait data. cos sin cos2 cos2 cos3 cos cos sin sin cos cos2 sin2 sin2 cos2 cos3 sin3 sin3 cos

11 The coefficients of Fourier series could be updated by solving the linear algebraic equation of (15). cos sin sin cos cos sin sin cos cos sin sin cos Where, and matrices are always invertible Continues-time extended kalman filter The continues-time extended kalman filter can be summarized as follows [13]. 1. The system equations are given as 15,,,, ~, h,,, ~, Calculate the following partial derivative of matrices: h h 3. Calculate the following matrices: 4. Kalman filter update equations:,,,h,, Where the nominal noise values and are zero. The following values are chosen for, and

12 Continues-time extended kalman filter is used to estimate the Fourier series coefficients. The coefficients represent the user activity mode. They are used as features and classified by k-nn classification algorithm. The coefficients for 4 standing, 9 ramp up and 1 ramp down trials are obtained and used as training set. Since, the human gait data for transient mode is not available; the classifier is trained with steady state values of coefficients. Thus, it must be noted that the training is merely done for the last 2 second at each activity mode. Then, knee and hip angle for standing, ramp up and ramp down are given to the system separately as test data. The coefficients are obtained for these test dataset at each time step. The activity mode is checked by k-nn classification method only every 1 steps. This is just done for simplification of simulation. Figure 11, 12 and 13 shows the satisfactory performance of kalman filter in tracking knee and hip angle at standing, ramp up and ramp down mode, respectively. k-nn classifier assigned the correct class to each walking mode for the last 2 seconds. It is completely reasonable, since the classifier is only trained for that period. Knee Angle Standing Mode Knee Estimated Hip Angle Hip Estimated Time (Sec) Figure 11.a: standing mode test data Ramp Up Walking Mode Standing Time Sample Figure 11.b: result of classifier

13 Knee Angle Ramp Up Mode Knee Estimated Hip Angle Time (Sec) Figure 12.a: ramp up mode test data Hip Estimated Ramp Down Walking Mode Ramp Up Standing Time Sample Figure 12.b: result of classifier

14 Knee Angle 1 5 Ramp Down Mode Knee Estimated Hip Angle Hip Estimated Time (Sec) Figure 13.a: ramp down mode test data Ramp Down Walking Mode Ramp Up Standing Time Sample Figure 13.b: result of classifier 4. Conclusion In the present project, a Fourier series is fitted into human gait data. The Fourier series coefficients are different at each activity mode. Thus, the difference is a valuable feature to identify user activity mode. Knee and hip angle are used for standing, level walking, ramp up and ramp down mode. An offline and online user activity recognition is developed. In offline activity recognition, a LM minimization problem is solved to obtain the optimum values for coefficients. Then, k-nn classification method used to classify the training coefficients. Each coefficient vector is related to an activity mode. Some test gait data applied to the trained system. The results verified the performance of method. However, the offline activity recognition is not useful for real time implementation. Therefore, the online activity recognition is proposed. In online activity recognition, to establish the model and obtaining the Fourier series coefficients appropriately; kalman filtering approach is used to estimate the coefficients. Then, the classification method is completely the same as before. Kaman based activity recognition resulted in satisfactory performance.

15 In this project, many other important human gait data such as GRF and hip torque were not available. In future work, these valuable information will be used. They will certainly increase the accuracy of user intent recognition. Moreover, the classifier will be trained for transient modes. Applying a more advanced classification method such as support vector machine (SVM) is in mind, too. Finally, it will be tried to implement the proposed user intent recognition. References [1] F. Zhang, H. Huang, Decoding movement intent of patient with multiple sclerosis for the powered lower extremity exoskeleton, Engineering in Medicine and Biology Society, Beskrivelse, pp , 213. [2] L. Peeraer, B. Aeyels and G. Van der Perre, Development of EMG-based mode and intent recognition algorithms for a computer-controlled above-knee prosthesis, Journal of Biomedical Engineering, vol. 12, pp , 199. [3] F. Zhang, H. Huang, Real Time Recognition of User Intent for Neural Control of Artificial Legs, Proceedings of the MyoElectric Controls/Powered Prosthetics Symposium Fredericton, New Brunswick, pp. 45 5, 211. [4] H. Varol, F. Sup and M. Goldfarb, Multiclass Real-Time Intent Recognition of a Powered Lower Limb Prosthesis, IEEE Transactions on Biomedical Engineering, vol. 57, no. 3, pp , 21. [5] J. R. Kwapisz, G. M. Weiss and S. A. Moore, Activity Recognition using Cell Phone Accelerometers, SensorKDD, Washington, pp , July 21. [6] V. D. Bogert, Kinematics, kinetics, amputee gait, [7] K. Levenberg, A method for the solution of certain problems in least squares, The Quarterly of Applied Mathematics, vol. 2, pp , [8] D. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrial and Applied Mathematics, vol. 11, pp , [9] G. Toussaint, Geometric proximity graphs for improving nearest neighbor methods in instance-based learning and data mining, International Journal of Computational Geometry and Applications, vol. 15, pp [1] J. Gou, T. Xiong and Y. Kuang, A Novel Weighted Voting for K-Nearest Neighbor Rule, Journal of Computers, vol. 6, no. 5, pp , 211. [11] T. M. Cover, and P. E. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory, vol. 13(1), pp , [12] R. Bitmead, A. Tsoi and P. Parker, A Kalman Filtering Approach to Short-Time Fourier Analysis, IEEE Transactions on Acoustic, Speech, and Signal Processing, vol. 34, no. 6, pp , [13] D. J. Simon, Nonlinear Kalman filtering, in Optimal State Estimation, 1 st ed. New Jersey, John Wiley & Sons, Inc., 26, 13, 13.2, pp