Gradient Compared l p -LMS Algorithms for Sparse System Identification

Transcription

1 Gradient Comared l -LMS Algorithms for Sarse System Identification Yong Feng,, Jiasong Wu, Rui Zeng, Limin Luo, Huazhong Shu. School of Biological Science and Medical Engineering, Southeast University, Nanjing 0096, China fengyong@seu.edu.cn. Laboratory of Image Science and echnology, the Key Laboratory of Comuter Netor and Information Integration, Southeast University, Nanjing 0096, China shu.list@seu.edu.cn Abstract: In this aer, e roose to novel -norm enalty least mean square (l -LMS) algorithms as sulements of the conventional l -LMS algorithm established for sarse adative filtering recently. A gradient comarator is emloyed to selectively aly the zero attractor of -norm constraint for only those tas that have the same olarity as that of the gradient of the squared instantaneous error, hich leads to the ne roosed gradient comared -norm constraint LMS algorithm (l GC-LMS). We exlain that the l GC-LMS can achieve loer mean square error than the standard l -LMS algorithm theoretically and exerimentally. o further imrove the erformance of the filter, the l NGC-LMS algorithm is derived using a ne gradient comarator hich taes the sign-smoothed version of the revious one. he erformance of the l NGC-LMS is suerior to that of the l GC-LMS in theory and in simulations. Moreover, these to comarators can be easily alied to other norm constraint LMS algorithms to derive some ne aroaches for sarse adative filtering. he numerical simulation results sho that the to roosed algorithms achieve better erformance than the standard LMS algorithm and l -LMS algorithm in terms of convergence rate and steady-state behavior in sarse system identification settings. Key Words: Least Mean Square Algorithm, Norm Constraint, Sarse System Identification, Ne Gradient Comarator INRODUCION Sarse systems, hose imulse resonses contain many near-zero coefficients and fe large ones, are very common in ractice. For examle, sarse ireless multi-ath channels, sarse acoustic echo ath, digital V transmission channels, and so forth. he least mean square (LMS) algorithm [] is the most idely used technique in alications lie system identification (SI) hich lays an imortant role in the alication of adative filtering. Hoever, the traditional LMS algorithm does not assume any structural information about the system to be identified and thus erforms oorly both in terms of steady-state excess mean square error (excess MSE) and convergence rate []. Recently there emerges a groing research interest in sarse system identification, for examle, sarse LMS algorithms ith different norm constraints, hich are mainly motivated by research of the least absolute shrinage and selection oerator (LASSO) [3] and comressive sensing (CS) [4]. he family of norm constraint LMS algorithms has become one of the main sarse LMS algorithms in adative filtering during the last fe years [5]. his or is suorted by the National Basic Research Program of China under Grant 0CB707904, by the NSFC under Grants 60344, 673, 30074, and by the SRFDP under Grants and , the Project-sonsored by SRF for ROCS, SEM, and by Natural Science Foundation of Jiangsu Province under Grant BK039 and by Qing Lan Project. Many norm constraint LMS algorithms have been roosed so far, for instance, l -norm enalty LMS (l -LMS) [, 6], l 0 -norm enalty LMS (l 0 -LMS) [7, 8] and l -norm enalty LMS (l -LMS) [9, 0], here the corresonding l, l 0 and l norms are incororated into the cost function of the standard LMS algorithm resectively, to increase the convergence seed and decrease the MSE as ell. Hoever, these algorithms aly the norm constraint of zero attractors to all the eight tas of the unnon system in general, leading to a sloing don of the convergence for those tas ho have the same olarity as the gradient of the squared error []. Fortunately, it can be imroved by using a gradient comactor that e ill shon in the later sections. o ic off the limitation above, e roose a gradient comared -norm enalty LMS algorithm (l GC-LMS) as ell as its imroved version the l NGC-LMS, as sulements of the conventional l -LMS algorithm. Numerical simulations sho that the roosed algorithms achieve better erformance than the standard LMS and l -LMS algorithms in sarse system identification settings. Simultaneously, aheri and Vorobyov have demonstrated that the l -LMS is suerior to other norm constraint LMS algorithms in convergence behavior under certain conditions [9]. he aer is organized as follos: In Section II, e roose the l GC-LMS and its extension the l NGC-LMS algorithm for sarse systems in details, after a brief revie of the standard LMS and l -LMS algorithms. hen the numerical simulations are given in Section III to investigate

2 the erformance of the to roosed algorithms. Finally, the aer is concluded in Section IV. PROPOSED ALGORIHMS hroughout this aer, matrices and vectors are denoted by boldface uer-case letters and boldface loer-case letters, resectively, hile variables and constants are in italic loer-case letters. he suerscrits ( ) reresents the transose oerators, and E[ ] denotes the exectation oerator.. Revieing standard LMS and l -LMS algorithms Let y be the outut of an unnon system ith an additional noise n at time, hich can be ritten as y x + n, () here the eight of length N is the imulse resonse of the unnon system, x reresents the inut vector ith covariance R, defined as x [ x, x,, x N+ ], and n is a stationary noise ith zero mean and variance σ. Given the inut x and outut y in an unnon linear system folloing the above settings, the LMS algorithm as roosed to estimate the eight vector decades ago. he cost function J of the standard LMS algorithm is defined as J e /, () here e y x denotes the instantaneous error and [[ ],[ ],...,[ ] ] N is the estimated eight of the system at time. Note that the / here is taen just for the convenience of comutation. hus, using gradient descent, the udate equation is ritten as J + μ + μex, (3) here μ is the ste size such that 0 < μ < λ max ith λ max being the maximum eigenvalue of R. For the sarse system identification in hich most of the tas in the eight vector are exactly or nearly zeros, the l -LMS [9] algorithm has been roosed ith the ne cost function J, ritten as J, e /+ γ, (4) here the norm is defined as N ( ) / [ ] i i ith 0 < <, and γ is a constant controlling the trade-off beteen the convergence seed and estimation error. hus, the udate of the l -LMS is then derived as sgn( ) + + μe ε + x, (5) here ρ μγ is an imortant arameter hich eights the -norm constraint and significantly affects the erformance of the algorithm, ε is a constant bounding the term and sgn(x) is the sign function, hich is zero for x 0, for x > 0 and - for x < 0. Comared to the standard LMS, the udate of the l -LMS has an extra udate term hich attracts all the coefficients in the eight to zeros and thus, accelerate the convergence. Hoever, it ill lead to a slo don for the convergence of those tas that are otimized by the term μex in the udate equation, since it does not distinguish the different sarsity levels of the system [].. Proosed l GC-LMS and l NGC-LMS algorithms In order to conquer the above limitation of the l -LMS, insired by [], e introduce a gradient comarator for the l -LMS to obtain a ne algorithm called the l GC-LMS, hich selectively emloys the zero attractor of norm on those tas that have the same olarity as the gradient of the squared instantaneous error. he udate equation for the l GC-LMS is given by sgn( ) + + μe ε + x here [ e x ] G, (6) G diag sgn( ) sgn / is a diagonal matrix. Furthermore, e also exlored an imroved version for the l GC-LMS, called l NGC-LMS, hich emloys a ne gradient comarator that selectively zero-attract only tas that have the same olarity as the gradient of the mean squared error. hus its udate is obtained as sgn( ) + + μe ε + x D, (7) here D sgn G i S i is the signed mean version of the G above, and it should be emhasized that the integer S (ractically set to 5 or more) emloyed here is significant hose logic ill be exlained belo. he roles of gradient comarators G and D are elaborated as follos: Let [ ] i be the ith element of the real eight of the unnon system, such that [ ] i > 0, [ ] i < 0 and [ ] i 0 are all the three ossibilities for the range of value [ ] i. Firstly, e consider the case [ ] i > 0 : () If [ ] i> [ ] i, then E[ ex] i E[( x + n x) x ] i < 0, and thus, ] i, i ith a high robability of at least 50% hich deends on the signal noise ratio (SNR), hereas, ] i, i holds u ith much higher robability due to the emloyed tric of signed mean method, hich taes the exectation to increase the lielihood of the equation ] i, i in this case or ] i, i 0 belo as much as ossible. Practically e set the value of S to be 5 or more to made it. () When [ ] i< [ ] i, if [ ] i< 0, e have ] i, i ith a high robability and ] i, i ith a much higher robability. Similarly, e also have ] i, i 0 and ] i, i 0 ith high and much higher robabilities resectively, if [ ] i> 0. Secondly, the same

3 logic reasoning ill also aly to the case [ ] i < 0. Finally, hen [ ] i 0, e have [ ] i> 0 or [ ] i< 0, ] i, i and ] i, i ith both high but different aforementioned robabilities are alays concluded. he l GC-LMS or l NGC-LMS algorithm ors in the same ay as the l -LMS hen ] i, i or ] i, i holds u, i.e., the norm constraint is emloyed to attract the tas to zeros in this case; And they ill be deduced to the standard LMS algorithm hen ] i, i 0 and ] i, i 0, hich neutralize the function of the zero attractor of the l -LMS. Additionally, it should be underlined that the global convergence and consistency of the l GC-LMS and l NGC-LMS remain roblematic, hich is inherited from the l -LMS hose cost function is not guaranteed to be convex. he Pseudo-codes for the l NGC-LMS algorithm is given in able. Furthermore, e can also derive some other gradient comared norm-constraint LMS algorithms including the l -LMS, l 0 -LMS and some of their variants. Added to this, it is also orth mentioning that the variable ste size (VSS) is non to have loer steady-state error as ell as faster convergence. hus, as a further extension, the VSS l GC-LMS and VSS l NGC-LMS can be easily derived in the same ay as it has been added to the -norm-lie LMS in [0]. 3 SIMULAIONS Numerical simulations are carried out for several scenarios in this section to investigate the erformances of the roosed l GC-LMS as ell as the l NGC-LMS algorithm in terms of the steady-state mean square deviation (MSD, defined as MSD E[ ] ) and convergence seed, and their results are comared ith the l -LMS and the standard LMS algorithm in the settings of sarse system identification ith different sarsity levels. Additionally, the suerior erformance of the l -LMS to other sarsity-aare modifications of LMS algorithms that are beyond the norm constraints, has been shon in Ref. [9], hich, to be brief, is not detailed here, nor is the contrast ith l GC-LMS in Ref. [] as a modification of l -LMS, due to the arameter deendence roblem in comarison ith the l -LMS algorithm [9]. 3. Examle : sarse system ith hite Gaussian inut In the first exeriment, e estimate a sarse unnon time-varying system of 6 tas ith, 4 or 8 tas that are assumed to be nonzeros, maing the sarsity ratio (SR) be /6, 4/6 or 8/6, resectively. he ositions of nonzero tas are chosen randomly and the values are s or - s randomly. Initially, e set SR /6 for the first 500 iterations, and after that e have SR 4/6 for the next 500 iterations, and then SR 8/6 for the last 500 iterations hich leaves a semi-sarse system. he inut signal and observed noise are both assumed to be hite Gaussian rocesses of length 56 ith zero mean and variances and 0.0, resectively, i.e., the signal noise ratio (SNR) is set to be 0 db. Other arameters are carefully selected as listed in able. Note that e use the same ste size μ for all the four filters and the same ρ, ε and for the three norm constraint LMS algorithms. All the simulations are averaged over 00 indeendent runs to smooth out their MSD curves. Fig. shos the MSD curves of the roosed l GC-LMS and l NGC-LMS algorithms ith resect to the number of iterations ith different sarsity levels, i.e., SR /6, 4/6 and 8/6 for the three iterative stages, comared ith the standard LMS and l -LMS algorithms. As shon in Fig., e can see that in the very sarse case (SR/6) for the first 500 iterations, the l NGC-LMS achieves the best erformance in terms of convergence seed and stable error, hile the l GC-LMS ors a little orse than the l -LMS, robably due to the aforementioned robability roblem of G, It still erform better than the standard LMS. With the sarsity decreasing (SR4/6, 8/6 for next stages ith 500 iterations each), the erformances of all the three -norm constraint LMS algorithms deteriorate as exected. Hoever, the l NGC-LMS still has better erformance than the l -LMS and the l GC-LMS, and the l GC-LMS still erforms better than the l -LMS in these to cases hen the system is semi-sarse. hus, it can be concluded that the roosed l GC-LMS and l NGC-LMS algorithms outerform the l -LMS for a system ith different sarsity levels and hite Gaussian inut. able. Pseudo-codes of the l NGC-LMS Algorithm Given Initial for end μ, ρ, N, S, ε,, x, L 0 zeros(n,),,yzeros(,l) i,,..., (L-N+) y x + n e y x sgn( ) + + μex D ε + G diag [ sgn( ex) sgn ]/ D sgn G i i S able. Parameters of the algorithms in the examle. Algorithms μ ρ ε S LMS [] l -LMS [9] [a] / [b] 0.5 l GC-LMS [c] l NGC-LMS 5 [a]~[c] are for SR/6, 4/6 and 8/6, resectively. Fig. shos the MSD curves of these algorithms tested ith resect to different sarsity levels (ith fixed ρ ), from hich e can see that the MSDs of all the three -norm constraint LMS algorithms increase ith the SR, hereas the MSD of the standard LMS is stable. As observed from Fig., the l NGC-LMS has the loest MSD hen the system is sarse, then comes by the l GC-LMS. It is imortant to note that the l NGC-LMS has loer MSD than the l -LMS hatever the sarsity level of the system is, under the common settings in this simulation. Generally, 0.5 is the default value for in most l -LMS related roblems ith fixed values, hereas e have exlored and achieved a ne l -LMS ith different methods of variable and better erformances, hich ill be secified in the folloing aer, due to the sace limitations.

4 able 3. Parameters of the algorithms in the examle. Fig.. MSD curves of different algorithms ith SNR 0 db and SR/6, 4/6 and 8/6, resectively.(hite Gaussian inut) Fig.. MSD curves of tested algorithms ith fixed ρ , different sarsity levels and SNR 0 db. 3. Examle : sarse system ith correlated inut he unnon system in the second examle is the same as the revious one, excet that e change the sitching times for different sarsity levels to the 3000th iteration and the 6000th iteration, resectively. he inut is no a correlated signal hich is generated by x + 0.8x + u and then normalized to variance, here u is a hite Gaussian noise ith variance 0 -. And the variance of the observed noise is set to 0 l in this examle. Other arameter choices for all the algorithms tested are listed in able 3. As it is set in the first examle, e choose the same μ, ρ, ε and for all the tested algorithms if they are required. Hoever, to solve the roblem underlined in the revious examle that the norm constraint LMS algorithms may reform orse than the standard LMS due to the choices of the arameter, different -norm constraint eight ρ's are selected for different sarsity levels in this case, hich ould utilize the sarse constraints better, and thus achieve loer steady-state MSD values than that of the standard LMS algorithm. In addition, all the MSD curves of the simulations are smoothed via 00 Monte-Carlo runs as ell. Algorithms μ ρ ε S LMS [] l -LMS [9] / 0.05 l [b] GC-LMS l NGC-LMS [c] 5 [a]~[c] are for SR/6, 4/6 and 8/6, resectively. Fig. 3 shos the MSD curves of the algorithms tested for different sarsity levels, i.e., SR /6 for the first 3000 iterations, SR 4/6 for the 300th to 6000th iteration and SR8/6 for the next 3000 iterations left. It can be seen from Fig. 3 that similar erformance trends are observed as in the first examle, i.e., most of the observations from Fig. also hold true for this case here the inut signal is related. For systems of different sarsity levels, the l NGC-LMS and l GC-LMS alays achieve the faster convergence and loer steady-state MSD than the l -LMS. Moreover, the l NGC-LMS alays erforms better than the l GC-LMS and then folloed by the l -LMS in any case. Hoever, as the sarsity ratio increases, the behaviors of all the norm constraint LMS algorithms tested in this aer hich are exected to be better than the standard LMS algorithm, deend strictly on the arameters ρ and the SNR of the inut. Secifically, as SR increases from /6 to 8/6, the smaller ρ, and / or the smaller SNR in a certain range, the better erformances hich the -norm constraint LMS algorithms achieve, and the smaller erformance gas beteen the l NGC-LMS and l GC-LMS, the l GC-LMS and l -LMS. Anyay, the l NGC-LMS erforms the best among these three -norm constraint LMS algorithms, and better than or equivalently to the standard LMS if roer arameters are selected as in the settings of this examle. Fig. 3. MSD curves of different algorithms ith SR/6 (ρ ), 4/6 (ρ ) and 8/6 (ρ 0-5 ), resectively. (correlated inut) 3.3 Examle 3: system of ECG-lie imulse resonse In this examle, e estimate a sarse system ith an ECG-lie imulse resonse (IR) of 56 tas, in hich 8 of them are nonzeros. Fig. 4 shos the ta vector of the system to be identified, hich is samled from an ECG signal and then rocessed into a simler version ith most of the small tas set to zeros, to mae it sarse for our simulations. he

5 inut signal and additional noise are hite Gaussian rocesses ith variance and 0., resectively. Other arameters are selected in able 4 and all the simulations are erformed 00 times. Note that e choose these arameters that are different from the revious to examles to yield better erformance in term of convergence seed and steady-state error lotted in MSD curves, hich is shon in Fig. 5. able 4. Parameters of the algorithms in the examle 3. Algorithms μ ρ ε S LMS l -LMS / l GC-LMS l NGC-LMS From Fig. 5, one can see that the revious observations still hold u in this much longer sarse system, the roosed l NGC-LMS and l GC-LMS outerform the l -LMS and standard LMS algorithm ith faster convergence seed and loer steady-state MSD. Fig. 4. he imulse resonse of the system in the examle 3. Fig. 5. MSD curves of different algorithms ith SNR 0 db and SR8/56. (ECG-lie system) 4 CONCLUSION We roose to gradient comared -norm enalty LMS algorithms, i.e., the l GC-LMS and l NGC-LMS algorithm, as extensions of the conventional l -LMS algorithm hich is established for sarse adative filtering recently. o gradient comarators are emloyed to selectively aly the zero attractor in l -LMS for only those tas that have the same olarity as that of the gradient of the squared instantaneous error or mean squared error. Numerical simulation results sho that the roosed algorithms yield better erformances than those of the standard LMS and l -LMS algorithms in sarse system identification settings in term of convergence seed and steady-state mean square deviation. Summarily, e can conclude from the above statements in this aer that the roosed l GC-LMS and l NGC-LMS are actually to aroaches to eight the -norm constraint for the l -LMS algorithm to select certain tas in order to erform better, and there are also some other methods to achieve this. herefore, our future or ill still focus on the choices of the arameters of the l -LMS algorithm for sarse system identification, including exloring the relationshi among the eight of norm constraint, the sarsity ratio of the imulse resonse and the signal-noise ratio of the inut, considering an adative arameter and ρ for different sarsity levels and / or different SNR, and develoing ne l -LMS algorithms for better identifying an unnon system hich is not assumed to be sarse or non-sarse, etc. REFERENCES [] B. Widro and S. D. Stearns, Adative signal rocessing, Engleood Cliffs, NJ, Prentice-Hall, Inc., 985. [] Y. L. Chen, Y.. Gu, and A. O. Hero, Sarse LMS for system identification, IEEE International Conference on Acoustics, Seech and Signal Processing, 009. [3] R. ibshirani, Regression shrinage and selection via the lasso: a retrosective. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol.73, No.3, 73-8, 0. [4] D. L. Donoho, Comressed sensing, IEEE rans. on Information heory, Vol.5, No.4, , 006. [5] B. K. Das, L. A. Azicueta-Ruiz, M. Charaborty, et al, A comarative study of to oular families of sarsity-aare adative filters, 4 th International Worsho on Cognitive Information Processing, 04. [6] K. Shi. and P. Shi, Convergence analysis of sarse LMS algorithms ith l -norm enalty based on hite inut signal, Signal Processing, Vol.90, No., , 00. [7] Y.. Gu, J. Jin, and S. L. Mei, l 0 Norm Constraint LMS Algorithm for Sarse System Identification, IEEE Signal Processing Letters, Vol.6, No.9, , 009. [8] G. L. Su, J. Jin, Y.. Gu, et al, Performance Analysis of l 0 Norm Constraint Least Mean Square Algorithm, IEEE rans. on Signal Processing, Vol.60, No.5, 3-35, 0 [9] O. aheri and S. A. Vorobyov, Sarse channel estimation ith l -norm and reeighted l -norm enalized least mean squares, IEEE International Conference on Acoustics, Seech and Signal Processing, 0. [0] M. L. Aliyu, M. A. Alassim, and M. S. Salman, A -norm variable ste-size LMS algorithm for sarse system identification, Signal, Image and Video Processing, -7, 04. [] B. K. Das and M. Charaborty, Gradient Comarator Least Mean Square Algorithm for Identification of Sarse Systems ith Variable Sarsity, APSIPA ASC, 0.