Kernel Adaptive Filtering Subject to Equality Function Constraints

Transcription

1 Kernel Adaptve Flterng Subject to Equalty Functon onstrants Badong hen Zhengda Qn Nannng Zheng José Príncpe Insttute of Artfcal Intellgence and Robotcs X'an Jaotong Unversty X'an hna Department of Electrcal and omputer Engneerng Unversty of Florda Ganesvlle FL 36 USA Abstract Kernel adaptve flters (KAFs) are powerful tools for onlne nonlnear system modelng whch are drect extensons of tradtonal lnear adaptve flters n kernel space wth growng lnear-n-the-parameters (LIP) structure However lke most other nonlnear adaptve flters the KAFs are "black box" models where no pror nformaton about the unknown nonlnear system s utlzed If some pror nformaton s avalable the "grey box" models may acheve mproved performance In ths work we consder the kernel adaptve flterng wth pror nformaton n terms of equalty functon constrants A novel Mercer kernel called the constraned Mercer kernel (MK) s proposed Wth ths new kernel we develop the kernel least mean square subject to equalty functon constrants (KLMS-EF) whch can satsfy the constrants perfectly whle achevng sgnfcant performance mprovement Keywords Kernel adaptve flterng; kernel least mean square; equalty functon constrants; I INTRODUTION Nonlnear system modelng fnds a wde range of applcatons n many real-world problems and s stll an actve area of research There are varous nonlnear adaptve flters that can be used for nonlnear system modelng among whch the kernel adaptve flters (KAFs) [] are very attractve because of ther desrable features such as convexty unversal approxmaton and onlne learnng manner The KAFs are developed by mplementng the well-establshed lnear adaptve flters n kernel space buldng a growng lnear-nthe-parameters (LIP) nonlnear model n the orgnal nput space They also belong to a class of learnng machnes called convex unversal learnng machnes (ULMs) [] So far many kernel adaptve flterng algorthms have been developed [3-9] among whch the kernel least mean square (KLMS) [4] s the smplest yet often most effectve one The KAFs are "black box" models and take no pror nformaton about the unknown system nto consderaton lke most other nonlnear adaptve flters In many cases however some pror knowledge about the unknown system s avalable and can be ncorporated nto the model ("grey box" model) to mprove the learnng performance A general way of expressng mathematcally the pror knowledge about the system s through some knds of constrants [0] In recent years a varety of constrants have been successfully ncorporated nto artfcal neural networks (ANNs) to acheve mproved learnng performance [0-5] In the present paper we propose to ncorporate some pror constrants nto the KAFs In partcular we consder ncorporatng the equalty functon constrants (EF) nto the KLMS algorthm A smlar approach has been appled n ANNs [0] As stated n [0] the equalty functon constrants have some advantages over other equalty constrants such as boundary value constrants (BV) [5] To solve the kernel adaptve flterng subject to equalty functon constrants we propose a novel Mercer kernel called n ths paper the constraned Mercer kernel (MK) whch s defned by multplyng the orgnal Mercer kernel by a weghtng functon correspondng to the sub-regons of the equalty functon constrants Usng ths new kernel we develop the KLMS subject to equalty functon constrants (KLMS-EF) Of course the proposed MK can also be appled to other KAFs and other kernel methods such as SVM The rest of the paper s organzed as follows In secton II we brefly ntroduce the KLMS algorthm and descrbe the learnng problem subject to equalty functon constrants In secton III we propose the constraned Mercer kernel and develop the KLMS-EF algorthm In secton IV we present smulaton results to demonstrate the desrable performance of the KLMS-EF Fnally n secton V we present our conclusons II KLMS AND EQUALITY FUNTION ONSTRAINTS A KLMS Gven a sequence of nput-output tranng examples u d m L where u U R d R wth U beng the nput doman our goal s to learn a nonlnear mappng f : U R that fts the data well under a specfc learnng crteron Under the mean square error (MSE) crteron ths learnng problem can be solved n an onlne manner (sample by sample) by usng the KLMS algorthm [4]: f0 0 e d f f f eu where f denotes an estmate of f at teraton e stands for the predcton error based on the last estmate f 0 denotes the step-sze parameter and s a reproducng Mercer kernel functon defned on U U e :U U R The KLMS algorthm () s actually the least mean square (LMS) ()

2 algorthm n kernel space derved by transformng the nput u nto the reproducng kernel Hlbert space (RKHS) H nduced by the Mercer kernel and applyng the LMS on the transformed data [4] The wdely adopted kernel s the Gaussan kernel: u u ( uu ) exp where 0 s the kernel bandwdth and denotes the Eucldean norm As one can see from () the KLMS creates a growng LIP nonlnear model where the nonlnear transformaton s determned by the selected Mercer kernel An appealng feature of the KLMS s that the lnear combnaton coeffcents are drectly related to the predcton errors B Equalty functon constrants The KLMS s a "black box" method wth whch the learned model s completely determned by the tranng data (assumng that the step-sze and kernel functon are gven) In ths paper some addtonal constrants wll be ncorporated nto the KLMS Specfcally consder the followng equalty functon constrants (EF) on the learned mappng f () f f u U U (3) where f s a pror known functon defned on U wth U beng the constrant doman a subset of the nput doman U The defnton doman of the functon f can be extended to the whole nput doman U by defnng Especally u f = f ( u ) st u argmn u c (4) u when u U cu Now our goal s to modfy the orgnal KLMS such that the learned model strctly satsfes the above constrants whle achevng mproved learnng performance A novel approach wll be proposed n the next secton to address ths ssue Note that n [0] the equalty functon constrants were successfully appled n an RBF model whch s mplemented n a batch mode (not an onlne manner) III ONSTRAINED MERER KERNEL AND KLMS-EF A onstraned Mercer kernel Defnton : The constraned Mercer kernel wth respect to the constrant doman U s defned by ( u u) ( u u) u uu (5) where ( uu ) s the orgnal Mercer kernel and ( u ) s a weghtng functon wth respect to the constrant doman U gven by ( u ) exp( ( u )) (6) where mn u c s the mnmal dstance from to the cu constrant doman U and 0 s a parameter for adjustng the slope of ( u ) Remark : ( u ) s a contnuous functon of u whch takes the value of zero when u U whle approachng0 when u s apart from the constrant doman U By defnton the constraned Mercer kernel ( uu ) wll gradually lose ts learnng capablty when u gets close tou Now we prove that ( uu ) s really a Mercer kernel functon over U U Obvously ( uu ) s a contnuous and symmetrc functon over U U So we only need to prove the postve-defnteness of ( uu ) For any n N and any choce of u u L un U and a a L an R we have N N N N a a ( u u ) a a ( u ) ( u ) ( u u ) j j j j j j j ( a) N j bb j ( u u j ) (7) 0 N where b a ( u ) b a ( u ) and (7) follows from the fact j j j that s a Mercer kernel Thus ( uu ) s postve-defnte and hence s also a Mercer kernel Remark : Though ( uu ) s postve-defnte t s not strctly postve defnte (SPD) snce we have ( uu ) 0 for any uu or u U B KLMS-EF To satsfy the equalty functon constrants the ntal estmate of f s set at (8) f ( ) ( ) ( ) 0 u u f u whch equals f when u U and approaches zero when u s apart from U Here f s gven by (4) to cover the whole nput doman Wth the above ntalzaton the proposed KLMS-EF algorthm becomes the KLMS wth the constraned Mercer kernel ( uu ) that s f0 () f() e d f f f ec u Remark 3: If f s a contnuous functon the learned mappng f s also a contnuous functon snce both ( u ) and ( uu ) are contnuous functons Remark 4: The computatonal complexty of the KLMS- EF s almost the same as that of the orgnal KLMS In addton there s only one extra free parameter n KLMS-EF (9)

3 namely the parameter whch controls the learnng rate and smoothness around the boundary of U Substtutng (5) nto the update rule of KLMS-EF we obtan f f e ( u ) () ( u ) (0) From (0) one can observe: ) f u U there s no update on f (also no dctonary update on the hdden nodes) ; ) f u s very close to U the update rate s very small because ( u ) 0 as ( ) 0 u learly the learned mappng at teraton s () f f e ( u ) ( u u) j j j j whch bascally conssts of two parts namely the pror known part f and the sequentally learned part j e ( u ) ( u u) The second part stll has a growng j j j lnear-n-the-parameters (LIP) structure lke usual KAFs although there s no dctonary update when u U To reduce the computatonal costs and memory requrements one can use some sparsfcaton technques [] or quantzaton methods [-4] to curb the network growth and obtan a compact model where U u u u u u The goal s to ft the functon based on the data and the equalty constrants We draw 000 tranng samples n whch 980 samples are drawn u u and 0 from the unform dstrbuton over samples equally spaced n U In addton the testng data contan 880 samples n whch 800 samples are drawn from u u and 80 samples the unform dstrbuton over equally spaced nu In the smulaton the tranng data are corrupted by addtve Gaussan nose wth zero mean and 005 standard devaton Fg shows the performance comparson between KLMS- EF ( ) and the orgnal KLMS wth dfferent step szes ( ) In both KLMS and KLMS-EF the kernel functon () s chosen as the Gaussan kernel wth bandwdth 04 At each teraton the testng mean square error (testng MSE) s computed on the testng set usng the flter resultng from the tranng set The plotted results are obtaned by averagng over 50 Monte arlo runs As one can see the equalty constrants can mprove the learnng performance and the KLMS-EF can outperform the KLMS wth dfferent step szes The testng outputs and desred responses n constrant doman are shown n Fg 3 It s evdent that the model traned by KLMS-EF fts the data much better than the model traned by KLMS IV SIMULATION RESULTS We present smulaton results to demonstrate the performance of the proposed KLMS-EF onsder the followng hyperbolod functon (as shown n Fg) [0] y u u u u () and the equalty functon constrants: f u u U (3) Fg onvergence curves n terms of the testng MSE Fg hyperbolod functon Fg3 Testng outputs (cross) and desred responses (crcle) n constrant doman: (a) KLMS; (b) KLMS-EF

4 Fg4 shows the performance of the KLMS-EF wth dfferent values of As we can see the leanng process s stopped when 0 In ths case we have ( u ) 0 and f f But the algorthm can work very well even wth a very small When s too large (say ) the performance wll deterorate In ths example the best performance s acheved at round In the smulaton the step szes are manually chosen such that all the ntal convergence speeds (except the case 0 ) are vsually smlar s an mportant parameter but t s relatvely easy to choose as n general a small value of wll brng satsfactory results Fg6 Testng MSEs versus dctonary szes Fg4 onvergence curves wth dfferent values of To curb the network growth one can use a quantzaton approach to develop the quantzed KLMS-EF (QKLMS-EF) algorthm (see [] for the detals about QKLMS) Fg5 llustrates the convergence performance of the QKLMS-EF wth dfferent quantzaton szes ( ) and Fg 6 shows the correspondng testng MSEs versus the dctonary szes Smlar to the QKLMS there s a trade-off between accuracy and dctonary sze for the QKLMS-EF Usually a larger quantzaton sze leads to a poorer accuracy but a smaller dctonary sze Wth a proper quantzaton sze however the algorthm can produce a small network whle achevng desrable performance V ONLUSION Kernel adaptve flters (KAFs) are powerful onlne learnng machnes But they are "black box" models and ther performance can be sgnfcantly mproved f some pror knowledge s ncorporated nto the learned models In ths study we developed an effcent kernel adaptve flterng algorthm by ncorporatng the equalty functon constrants nto the kernel least mean square (KLMS) algorthm A novel Mercer kernel called the constraned Mercer kernel (MK) was proposed The kernel least mean square subject to equalty functon constrants (KLMS-EF) was then developed wth ths new kernel whch can satsfy the constrants perfectly whle achevng sgnfcant performance mprovement Smulaton results confrmed the excellent performance of the new algorthm AKNOWLEDGMENTS Ths work was supported by 973 Program (No 05B35703) and Natonal NSF of hna (No 6375) REFERENES Fg5 onvergence curves wth dfferent quantzaton szes [] W Lu J Prncpe S Haykn Kernel Adaptve Flterng: A omprehensve Introducton Wley 00 [] J Prncpe B hen Unversal Approxmaton wth onvex Optmzaton: Gmmck or Realty? IEEE omputatonal Intellgence Magazne vol 0 no pp [3] J Kvnen A J Smola and R Wllamson Onlne learnng wth kernels IEEE Trans Sgnal Process vol 5 no 8 pp Aug 004 [4] W Lu P Pokharel J Prncpe The kernel least mean square algorthm IEEE Transactons on Sgnal Processng vol 56 pp [5] W Lu J Prncpe Kernel affne projecton algorthm EURASIP J Adv Sgnal Process vol 008 Artcle ID 7849 pages do: 055/008/7849 [6] Y Engel S Mannor R Mer The kernel recursve least-squares algorthm IEEE Transactons on Sgnal Processng vol 5 pp

5 [7] W Lu Il Park Y Wang J Prncpe Extended kernel recursve least squares algorthm IEEE Transactons on Sgnal Processng vol 57 pp [8] Rchard J Bermudez P Honene Onlne predcton of tme seres data wth kernels IEEE Transactons on Sgnal Processng vol 57 pp [9] M Yukawa Multkernel adaptve flterng IEEE Transactons on Sgnal Processng vol 60 pp [0] K Slavaks S Theodords and I Yamada Onlne kernel-based classfcaton usng adaptve projecton algorthms IEEE Trans Sgnal Process vol 56 no 7 pp Jul 008 [] F Orabona J Keshet B aputo Bounded kernel-based onlne learnng Journal of Machne Learnng Research vol 0 pp [] B hen S Zhao P Zhu J Prncpe Quantzed kernel least mean square algorthm IEEE Transactons on Neural Networks and Learnng Systems vol 3 no pp -3 0 [3] B hen S Zhao P Zhu J Prncpe Quantzed kernel recursve least squares algorthm IEEE Trans On Neural Networks and Learnng Systems vol 4 no [4] Xu X Qu H Zhao J Yang X hen B Quantzed kernel least mean square wth desred sgnal smoothng Electroncs Letters vol 5 no8 pp [5] B hen S Zhao P Zhu J Prncpe Mean square convergence analyss for kernel least mean square algorthm Sgnal Processng vol 9 pp [6] S Zhao B hen J Prncpe Kernel adaptve flterng wth maxmum correntropy crteron n Proceedngs of Internatonal Jont onference on Neural Networks (IJNN) San Jose alforna USA pp [7] Bhen Z Yuan N Zheng J Prncpe Kernel mnmum error entropy algorthm Neurocomputng vol [8] Sade R Lengelle P Honene Rchard R Achkar Nonlnear adaptve flterng usng kernel based algorthms wth dctonary adaptaton Internatonal Journal of Adaptve ontrol and Sgnal Processng 05 Publshed onlne n Wley Onlne Lbrary (wleyonlnelbrarycom) do: 000/acs548 [9] T K Paul T Ogunfunm A kernel adaptve algorthm for quaternonvalued nputs IEEE Transactons on Neural Networks and Learnng Systems vol 6 no [0] L ao and B Hu Generalzed constrant neural network regresson model subject to equalty functon constrants Proceedngs of Internatonal Jont onference on Neural Networks (IJNN) pp [] O L Mangasaran and E W Wld "Nonlnear knowledge n kernel approxmaton" IEEE Transactons on Neural Networks vol 8 no pp [] Y Qu and BG Hu "Generalzed constrant neural network regresson model subject to lnear prors" IEEE Transactons on Neural Networks vol no pp [3] K S McFall and J R Mahan "Artfcal neural network method for soluton of boundary value problems wth exact satsfacton of arbtrary boundary condtons" IEEE Transactons on Neural Networks vol 0 no 8 pp [4] S hen X Hong and J Harrs "Grey-box radal bass functon modellng" Neurocomputng vol 74 no 0 pp [5] X Hong and S hen "A new RBF neural network wth boundary value constrants" IEEE Transactons on Systems Man and ybernetcs Part B: ybernetcs vol 39 no pp