The content is identical to the content of the published paper, but without the final typesetting by the publisher.

Transcription

1 This documet cotais the publisher s pdf-versio of the refereed paper: Efficiet Trackig of the Domiat Eigespace of a Normalized Kerel Matrix by Geert Gis, Ilse Smets, ad Ja Va Impe which has bee archived o the uiversity repository Lirias ( of the KU Leuve. The cotet is idetical to the cotet of the published paper, but without the fial typesettig by the publisher. Whe referrig to this work, please cite the full bibliographic ifo: Geert Gis, Ilse Smets, Ja Va Impe (2008). Efficiet Trackig of the Domiat Eigespace of a Normalized Kerel Matrix. Neural Computatio, 20(2): The joural ad the origial published paper ca be foud at: The correspodig author ca be cotacted for additioal ifo. Coditios for ope access are available at:

2 LETTER Commuicated by Léo Bottou Efficiet Trackig of the Domiat Eigespace of a Normalized Kerel Matrix Geert Gis geert.gis@cit.kuleuve.be Ilse Y. Smets ilse.smets@cit.kuleuve.be Ja F. Va Impe ja.vaimpe@cit.kuleuve.be Bioprocess Techology ad Cotrol, Katholieke Uiversiteit Leuve, W. de Croylaa 46, B-3001 Leuve, Belgium Various machie learig problems rely o kerel-based methods. The power of these methods resides i the ability to solve highly oliear problems by reformulatig them i a liear cotext. The domiat eigespace of a (ormalized) kerel matrix is ofte required. Ufortuately, the computatioal requiremets of the existig kerel methods are such that the applicability is restricted to relatively small data sets. This letter therefore focuses o a kerel-based method for large data sets. More specifically, a umerically stable trackig algorithm for the domiat eigespace of a ormalized kerel matrix is proposed, which proceeds by a updatig (the additio of a ew data poit) followed by adowdatig(theexclusioofaolddatapoit)ofthekerelmatrix. Testig the algorithm o some represetative case studies reveals that a very good approximatio of the domiat eigespace is obtaied, while oly a miimal amout of operatios ad memory space per iteratio step is required. 1 Itroductio Kerel-based methods have become icreasigly popular i various machie learig problems, such as classificatio, patter recogitio, fuctio estimatio, system idetificatio, ad sigal processig problems (Schölkopf & Smola, 2002; Suykes, Va Gestel, De Brabater, De Moor, &Vadewalle,2002).Themaiadvatageofthesekerelmethodsisthat they are capable of solvig highly oliear problems by exploitig liear techiques, which is accomplished by implicitly mappig all data poits i a ofte high-dimesioal space. The majority of kerel-based methods, such as kerel pricipal compoet aalysis (KPCA; Schölkopf, Smola, & Müller, 1998), fixed-size, Neural Computatio 20, (2008) C 2008 Massachusetts Istitute of Techology

3 524 G. Gis, I. Smets, ad J. Va Impe least-squares support vector machies (FS-LSSVM; Suykes et al., 2002), ad spectral clusterig (Weiss, 1999; Ng, Jorda, & Weiss, 2002), rely, either directly or idirectly, o the eigevectors of the symmetric kerel Gram matrix K, which provides a similarity measure for the available data poits. For most applicatios, oly the domiat m eigevectors of the kerel matrix K are eeded, where m is much smaller tha. The eigespace of the kerel matrix is most ofte computed by meas of sigular value decompositio (SVD). 1 Although the SVD has a high umerical precisio ad stability, a substatial drawback is represeted by the computatioal demads: typically O( 3 ) operatios with a O( 2 )memory requiremet, or O( 2 m) for deflatio schemes (Golub & Va Loa, 1996). Furthermore, the mappig (or embeddig) is obtaied oly for existig data poits, such that the computatios must be performed all over agai for every ew data poit. The applicability of these otherwise well-performig kerel methods is therefore limited to relatively small data sets ad off-lie problems. Algorithms that try to alleviate these problems are beig developed (Rosipal & Girolami, 2001; Kim, Fraz, & Schölkopf, 2003) but to the best of our kowledge, a truly olie algorithm has ot bee reported so far. Hoegaerts, De Lathauwer, Suykes, ad Vadewalle (2004, 2007) proposed a algorithm to track the domiat subspace for solvig KPCA problems quasi-olie. Based o the SVD of the kerel matrix K, their algorithm solves the problem of the matrix expadig i both row ad colum dimesio whe a ew poit is added to the data set, a situatio SVD updatig schemes are uable to hadle. This method is, however, ot applicable to ormalized kerels, which are ofte used i spectral clusterig (Weiss, 1999; Ng et al., 2002) ad yield better results tha oormalized kerels. Ispired by the algorithm of Hoegaerts et al. (2004, 2007), this letter proposes a algorithm to track the eigespace of a ormalized kerel with satisfyig accuracy, while requirig oly O(m 2 ) operatios ad O(m) memory space. The algorithm starts by updatig the kerel matrix (the additio of a ew data poit) after which a dowdatig (the exclusio of a old data poit) is performed. Sectio 2 gives a short descriptio of the Laplacia kerel matrix. Sectio 3 gives a detailed descriptio of the proposed trackig algorithm. The umerical stability of the algorithm is discussed i sectio 4, ad its performace o two bechmark data sets is assessed i sectio 5. Fially, coclusios are draw i sectio 6. 1 Because the kerel matrix is positive (semi-)defiite, the sigular value decompositio (SVD) is idetical to a ormal eigevalue decompositio (EVD). SVD is preferred over EVD because of its umerical precisio ad stability.

4 Trackig the Domiat Normalized Kerel Eigespace TheLaplaciaKerelMatrix Give a data set S ={x 1, x 2,...,x },withx i R p, the symmetric kerel Gram matrix K is defied as [K ij = k(x i, x j ), (2.1) where the kerel fuctio k provides a pairwise similarity measure betwee data poits. The most commo choice for this kerel fuctio is the expoetial (or gaussia, or radial basis fuctio) kerel, give by k E (x i, x j ) = exp ( x i x j 2 ). (2.2) σ 2 Whe workig with a ormalized kerel, as is ofte doe i, for example, spectral clusterig, a trasformatio is applied to the Gram matrix before the leadig eigevectors are computed. A example of such a trasformatio is divisive ormalizatio, which uses the Laplacia L of the kerel matrix K (Weiss, 1999; Ng et al., 2002): L = D 1 2 KD 1 2 (2.3) { 0 if i j [D ij = k=1 [K ik if i = j. (2.4) Notice that the ormalizatio matrix D is sometimes defied slightly differetly: [D ii = k i [K ik.thisdefiitioofteleadstobetterembeddig of the data poits ad has o sigificat impact o the algorithm preseted i sectios 3.1 ad TrackigAlgorithm Whe trackig the domiat eigespace of the Laplacia kerel matrix, ew poits are added to the data set ad old poits are discarded. Ispired by the algorithm of Hoegaerts et al. (2004, 2007), the algorithm proposed i this sectio is a combiatio of a update step, i which a ew poit is added to the data set, ad a dowdate step, where a old poit is removed from the data set. 3.1 Updatig the Eigespace. Whe a ew data poit is added to the data set, the ew ( + 1) m eigespace U (+1)m of the ew Laplacia L +1 must be computed. While the eigespace ca be calculated directly from the ew data set usig batch SVD calculatios, this computatio typically requires O( 3 ) operatios ad becomes iefficiet ad slow as icreases.

5 526 G. Gis, I. Smets, ad J. Va Impe Istead of performig the calculatios from scratch, a approximatio of the desired eigespace ca be computed requirig oly O(m 2 ) operatios, assumig that the m domiat eigespace U m ad correspodig m m diagoal eigevalue matrix of L are kow, together with the diagoal ormalizatio matrix D. The update of the oormalized kerel matrix K is located o the last row ad colum of the matrix, so the ew oormalized kerel matrix K +1 ca be writte as [ K a u K +1 =, (3.1) a T u b u where the 1vectora u ad the scalar b u are defied as [a u i = k(x i, x +1 ), (3.2) b u = k(x +1, x +1 ). (3.3) The oormalized kerel matrix K ca be computed from the ormalized kerel matrix L usig equatio 2.3, provided the ormalizig matrix D is kow: K = D 1/2 L D 1/2. (3.4) Substitutig equatio 3.4 i 3.1 yields K +1 = [ 1/2 D L D 1/2 a T u a u b u. (3.5) This matrix ca the be factorized as K +1 = [ D 1/2 0 0 T 1 [ L D 1/2 a T u D 1/2 b u a u [ D 1/2 0 0 T 1, (3.6) where 0 is a 1 ull vector. Whe D +1 is block-partitioed i its upper block D +1,U ad the scalar d +1,L, D +1 = [ D+1,U 0 0 T d +1,L, (3.7)

6 Trackig the Domiat Normalized Kerel Eigespace 527 equatio 3.6 ca be substituted i equatio 2.3 to obtai a recursive expressio for the ew Laplacia matrix L +1, L +1 = [ D 1/2 +1,U 0 0 T d 1/2 +1,L [ D 1/2 0 0 T 1 [ D 1/2 [ D 1/2 +1,U 0 0 T T d 1/2 +1,L [ L D 1/2 a T u D 1/2 b u a u.... (3.8) Whe the diagoal structure of both D +1,U ad D is exploited, the pread postmultiplicatio ca easily be reformulated as L +1 = D 1/2 +1 D+1 = [ L D 1/2 a T u D 1/2 [ D 1 +1,U D 0 b u 0 T d 1 +1,L a u D 1/2 +1, (3.9). (3.10) The matrix D 1 +1,U D is diagoal, with the diagoal elemets equal to [ D 1 +1,U D = ii d ii,, (3.11) d ii,+1 where d ii, ad d ii,+1 are the ith diagoal elemets of D ad D +1, respectively. If the umber of data poits is large eough, d ii, ad d ii,+1 ca be expressed as d ii, = d ii (3.12) d ii,+1 ( + 1) d ii, (3.13) where d ii is defied as the average of d ii over all (or 1) data poits. 2 Based o these relatios, equatio 3.11 becomes D 1 +1,U D + 1 I. (3.14) 2 I work with the alterative ormalizatio [D ii = i k K ik,theaveraged must be defied over 1 data poits. However, the results remai virtually uchaged, with ( 1)/ beig used istead of /( + 1) i further steps of the algorithm.

7 528 G. Gis, I. Smets, ad J. Va Impe Substitutig equatio 3.14 i 3.10 yields D I 0 T 0 1 d +1,L. (3.15) Because the cetral matrix i equatio 3.9 still cotais the ukow matrix L, its kow rak-m approximatio L U m Um T is used, i accordace with Hoegaerts et al. (2007): L +1 D 1/2 +1 [ U 0 A [ Um U 0 = A= 0 T m D 1/2 [ [ O m U T 0 O T m a u D 1/2 b u b u 1 0 D u = a u 2 1 D u A T D 1/2 +1 (3.16) (3.17) (3.18). (3.19) Here, O m is a m 2 ull matrix. I the ext step, the matrix D +1 is folded ito the eigespace [U 0 A by usig equatio 3.15: D 1/2 +1 [ U 0 A = [ U0 A (3.20) U0 + 1 U m = + 1 U 0 (3.21) A 0 T m + 1 D 1/2 a u + 1 D 1/2 b u b u 2 d +1,L 2 d +1,L a u. (3.22)

8 Trackig the Domiat Normalized Kerel Eigespace 529 A ew eigespace [U0 A is hece obtaied. As ca be see from equatio 3.21, the foldig of D+1 has o ifluece o the orthogoality of the m eigevectors i U0, oly alterig their orm.3 The ( + 1) 2 matrix A cotais two vectors, which are ot orthogoal to the eigespace. Therefore, A is decomposed ito a orthogoal (A ) ad a parallel compoet (A )withrespecttothespacespaedbyu 0 (ad thus U 0), i a way similar to Hoegaerts et al. (2004, 2007): A = A + A (3.23) = ( I +1 U 0 U0 T ) A + U 0 U0 T A (3.24) = Q A R A + U 0 U T 0 A. (3.25) The QR-factorizatio of A = Q AR A has bee used to make the colums of A mutually orthogoal, resultig i the ( + 1) 2 matrix Q A ad the 2 2 matrix R A. The computatio of Q A ad R A requires oly a affordable umber, that is, O(m), of operatios (Golub & Va Loa, 1996). The updated eigespace ca be obtaied usig the decompositio: [ [ U0 A = U I m + 1 Q A + 1 Om T U T 0 A R A. (3.26) Usig equatio 3.21, the defiitio of U0,wecarewritethismatrixas + 1 I m U [ U0 A 0 T + 1 [ U A 0 Q A (3.27) + 1 Om T R A = + 1 Q u R u. (3.28) Substitutig equatio 3.28 ito 3.16 yields ( [ ) L Q O u R u Ru T Qu T, (3.29) O T D u where O is a 2 matrix of zeroes. 3 Strictly speakig, U0 does ot cotai eigevectors because U T 0 U 0 I.Thisissue will be addressed i a later stage of the algorithm by itroducig the same factor i the QR-decompositio of A.

9 530 G. Gis, I. Smets, ad J. Va Impe Fially, the eigevalue decompositio of the three middle matrices is performed usig the SVD, 4 costig O(m 3 ) operatios: R u [ O T O D u R T u = V +1 V T. (3.30) The last two eigevectors, correspodig to the smallest two eigevalues, are the discarded, ad the factor /( + 1) is icluded i the cetral matrix. This yields the fial result, L +1 U (+1)m +1 U T (+1)m, (3.31) where the ( + 1) m updated eigespace is obtaied as U (+1)m = Qu V, (3.32) together with the m m eigevalue matrix: +1 = (3.33) This updated eigespace U (+1)m is obtaied i O(m 2 ) operatios, ad the updatig of the eigevalues +1 requires the egligible amout of O(m) operatios. Hece, the global algorithm is capable of computig the embeddig of a ew data poit i O(m 2 ) operatios while usig O(m) memory space. 3.2 Dowdatig the Eigespace. With each update of the eigespace, its dimesio icreases, resultig i higher memory requiremets ad slower computatios. Eve with the updatig scheme proposed i sectio 3.1, this is impractical i olie applicatios ad subspace trackers, where the dimesio of the studied eigespace eeds to remai costat. Therefore, every update of the data set ad its kerel eigespace must be followed by a dowsizig, durig which the oldest data poit is removed from the data set. 4 SVD ad EVD are equal oly if the matrix is positive (semi-)defiite. This requiremet ca be relaxed by usig the properties λ(a) + x = λ(a + xi )adu(a) = u(a + xi ) to lift the matrix ito the positive semidefiite regio before performig the SVD.

10 Trackig the Domiat Normalized Kerel Eigespace 531 As i sectio 3.1, the oormalized kerel matrix K +1 is partitioed. This time, the first row ad first colum are separated from the rest of the matrix: [ bd a T d K +1 =. (3.34) a d K K is the oormalized kerel matrix of the data set, cotaiig oly the last data poits. The 1vectora d ad scalar b d cotai all iformatio pertaiig to the first data poit, which is to be elimiated. Whe pre- ad postmultiplyig K +1 with a expaded versio of D, the rescalig matrix for the last data poits, yields the matrix L : L [ α 0 T = 0 D 1/2 K +1 [ α 0 T 0 D 1/2 Whe equatio 3.34 is used, this expressio becomes L = [ α 2 b d αd 1/2 αa T d D 1/2 a d D 1/2 K D 1/2. (3.35). (3.36) As ca clearly be see, L cotais the dowdated Laplacia L o its last rows ad colums. The parameter α ca be freely chose because it iflueces oly the first row ad colum of L. D +1 is block-partitioed i the scalar d +1,U ad the lower block D +1,L, [ d+1,u 0 T D +1 =. (3.37) 0 D +1,L Whe equatio 3.4 is substituted i equatio 3.35, the matrix L becomes L = [ α 0 T 0 D 1/2 D 1/2 +1 L +1 D 1/2 +1 [ α 0 T 0 D 1/2. (3.38) Because all matrices i equatio 3.38 except L +1 are diagoal, L ca be reformulated as L = D 1/2 L +1 D 1/2 (3.39) [ D αd+1,u 0 T =. (3.40) 0 D 1 D +1,L

11 532 G. Gis, I. Smets, ad J. Va Impe With the parameter α selected to be equal to +1 /d +1,U ad uder the assumptio also made i sectio 3.1, that is large eough to drive d ii,+1 /d ii, toward the ( + 1)/ ratio, 5 D ca be approximated as D + 1 I +1. (3.41) Substitutig equatio 3.41 i 3.39 ad performig the SVD of L +1 results i L + 1 U (+1)m +1 U T (+1)m. (3.42) To obtai the dowdated Laplacia L from L,thefirstrowofU (+1)m, represeted by the 1 m vector u T must be discarded, retaiig U m : L [ u T U m +1 [ u U T m [ u T +1 u u T +1 U T m U m +1u U m +1U m T (3.43). (3.44) Thus, a approximate eigevalue decompositio of L is obtaied: L + 1 U m +1U T m. (3.45) The colums of U m are ot mutually orthoormal. Istead of performig a expesive QR-decompositio of U m to restore orthoormality, a trasformatio matrix M is used, as proposed by Hoegaerts et al. (2004, 2007). Thism m trasformatio matrix M is defiedas 1 M = 0 T [ [ u u 1 u 2 m 1 u = 1 u 2 0 m 1 I m 1 u, (3.46) where u = u/ u is the vector u divided by its legth. The m (m 1) matrix u is the right ull space of u T ad is obtaied i O(m 3 ) operatios. 5 Or toward /( 1), depedig o the defiitio of the ormalizatio matrix D.

12 Trackig the Domiat Normalized Kerel Eigespace 533 Whe the particular structure of the trasformatio matrix M is exploited, its iverse M iv ca be computed quickly: [ u T 1 u 2 M iv = M 1 =. (3.47) u T The matrix +1 is adjusted to compesate for the effects of this orthoormalizatio trasformatio ad subsequetly decomposed ito its eigevalues ad eigevectors: M iv +1 M T iv = U UT. (3.48) Fially, this yields the eigevalue decompositio of L : L U m U T m, (3.49) where the m dowdated eigespace is defied as U m = U m MU (3.50) ad the m m dowdated eigevalue matrix as = + 1. (3.51) The domiat term i the dowsizig cost is the M-trasformatio, which requires O(m 2 ) operatios. Whe the dowsizig is preceded by a updatig, the SVD of the updatig step may be postpoed util after dowdatig. 3.3 Trackig the Eigespace. Whe the updatig scheme from sectio 3.1 is combied with the dowdatig scheme from sectio 3.2, a trackig algorithm is obtaied. This algorithm is able to track the domiat eigespace of the ormalized Laplacia kerel, requirig oly O(m 2 ) operatios ad O(m) memory space per iteratio step. The complete trackig algorithm is summarized i Table 1. The proposed algorithm also applies to the trackig of the domiat eigespace of a oormalized kerel matrix by simply makig the followig assumptios, + 1 lim = 1, (3.52) lim [D ii = 1, (3.53)

13 534 G. Gis, I. Smets, ad J. Va Impe Table 1: Overview of the Laplacia Eigespace Trackig Algorithm. Give (i) D,thedivisiveormalizatiomatrixofK, beig a kerel matrix associated with the data poits x k,...,x k+ 1 ad (ii) U m ad,themleadig eigevectors ad correspodig eigevalues of the ormalized kerel matrix L = D 1/2 K D 1/2 of K, calculate (i) D,thedivisiveormalizatiomatrixofK, beig a kerel matrix associated with the data poits x k+1,...,x k+ ad (ii) U m ad,themleadig eigevectors ad correspodig eigevalues of the ormalized kerel matrix L = D 1/2 by Updatig: 1. D = 2. d +1,L = a T u 1 + b u [ D 1 + a u a T u 1 + b u [ Um 3. U 0 = 0m T D u = A + 1 D 1/2 a u + 1 D 1/2 a u = b u b u 2 d +1,L 2 d +1,L ( ) QR 6. Q A R A I +1 U 0 U0 T A 7. Q u = [U 0 Q A + 1 I m U0 T A 8. R u = + 1 Om T R A [ 9. u = + 1 R O u R O T u T D u Dowdatig: [ 0 T 1. D 1 = [ 0 I D +1 1 a d I 2. [ u T U m Q u K D 1/2 of K

14 Trackig the Domiat Normalized Kerel Eigespace 535 Table 1: (Cotiued) [ u 3. M = 1 u 2 u 4. M iv = 5. Q d = U m M 6. d = +1 [ u T 1 u 2 u T M iv u M T iv Performig a SVD ad a rotatio: 1. U UT SVD,m rak d 2. U m = Q du ad adjustig the formulas accordigly (thus recoverig the algorithm of Hoegaerts et al., 2007). 4 Numerical Stability Although the proposed trackig algorithm is aalytically soud, umerical aspects might limit its applicability. Therefore, the umerical stability of all steps of the algorithm, as listed i Table 1, is ivestigated. 4.1 Computatio of A. The computatio of A i the fifth step of the updatig part of the trackig algorithm is potetially ustable because it requires the iversio of d +1,L i the bottom row. Whe equatios 2.1 ad 2.4, are used, this scalar d +1,L ca be explicated as +1 d +1,L = k(x i, x +1 ) (4.1) i=1 = k(x +1, x +1 ) + k(x i, x +1 ). (4.2) i=1 Thus, d +1,L is lower bouded by k(x +1, x +1 ), the self-similarity of the data poit x +1. Cosequetly, the iversio of d +1,L i the updatig scheme will be stable (1) whe this self-similarity is sufficietly large or (2) whe the ew data poit x +1 is located sufficietly close to the old data poits x i such

15 536 G. Gis, I. Smets, ad J. Va Impe that i k(x i, x +1 ) becomes large eough to drive d +1,L away from zero. 6 This last coditio should always be satisfied i practice, as x +1 ot beig sufficietly similar to the other data poits idetifies it as a outlier uder the chose kerel parameters. For the expoetial kerel, give by equatio 2.2, the self-similarity ca easily be quatified: k E (x +1, x +1 ) = 1. (4.3) Therefore, the expoetial kerel will ever suffer from umerical istabilities i the updatig step. 4.2 Projectio of A. A secod potetially ustable step i the algorithm provided i Table 1 is the projectio of A oto the space spaed by U 0 ad the subsequet QR-decompositio of A. If the two vectors cotaied i A are (early) colliear, oly the first of the two colums of Q will be computed accurately, while the secod colum will be heavily iflueced by umerical imprecisios. Whe this pheomeo occurs, the correspodig diagoal elemet of R will also be very small. Therefore, oe must check (1) whether the secod colum of Q is perpedicular to the space spaed by the colums of U 0,ad/or(2)whether the bottom right elemet of R is sufficietly large. If either of these two coditios is violated, the umerical accuracy is improved sigificatly by settig the secod colum of Q ad bottom right elemet of R equal to Computatio of M ad M iv. The computatio of the orthogoality trasformatio matrix M ad its iverse M iv is directly depedet o the orm of the vector u. Because u is upper bouded by 1, oly the limit cases where u is close to either 1 or 0 must be cosidered. As discussed by Hoegaerts et al. (2007), the umber of tracked compoets should be recosidered whe u is (early) equal to 1. I this case, the trasformatio matrix M should be set equal to u. Whe u approaches 0, the computatio of u might become umerically ustable, leadig to large relative errors. Therefore, it is recommeded to restore the orthogoality of U m by performig a full QR-decompositio istead of usig the orthogoality trasformatio matrix M wheever u is too small. This slightly icreases the computatioal requiremets, but the whole algorithm still requires O(m 2 ) operatios. 6 This is also a requiremet for umerical stability whe usig the alterative ormalizatio.

16 Trackig the Domiat Normalized Kerel Eigespace Eigevector Orthogoality. Whe the domiat eigespace of the kerel matrix is tracked, umerical errors might lead to the loss of orthogoality betwee the eigevectors. Here, the orthogoality measure, E = I m Ũ T Ũ 2, (4.4) is moitored. A value close to 0 idicates good orthogoality (ad orthoormality) of the eigespace. Wheever E exceeds a certai threshold, reorthogoalizatio is performed (e.g., by QR-decompositio), resultig i a more robust algorithm. 5 PerformaceAssessmet To properly assess the practical applicability ad performace of the proposed algorithm, umerical tests are executed o two bechmark data sets. I each test case, the performace of the proposed trackig algorithm is compared with that of other partial eigevalue problem algorithms, used for batch computatios. The bechmark data sets used i this performace assessmet are discussed i sectio 5.1 ad the referece batch algorithms i sectio 5.2. The mathematical performace measure is provided i sectio 5.3. The umerical results of the compariso are detailed i sectios 5.4, 5.5, ad 5.6. Fially, the bechmark s computatioal aspects are discussed i sectio Bechmark Data Sets. The performace of the proposed updatig algorithm is tested o two bechmark data sets. The first data set cosists of 7500 data poits of dimesio 2, sampled from oe of four prototypes located o (1, 0), ( 1, 0), (0, 1), ad (0, 1) with N (0, 1 4 )gaussiaoise. Part of this data set is depicted i Figure 1. The secod data set is the well-kow abaloe bechmark data set (Blake & Merz, 1998) with 4177 data poits of dimesio 7. I both cases the radial basis kerel fuctio is used: k(x i, x j ) = exp ( x i x j 2 ). (5.1) σ 2 The kerel width σ is equal to 1 for the artificial data set ad equal to 10 for the abaloe data set. The ormalizatio is performed usig the stadard defiitio: [D ii = i [K ik. The eigespectra of the kerel matrices are depicted i Figure 2. For the artificial data set, the first 4 eigevalues, required to classify the data ito the origial clusters, capture 82% of the total spectrum; the first 10 eigevalues

17 538 G. Gis, I. Smets, ad J. Va Impe Figure 1: The artificial data set used for testig the updatig algorithm. capture over 96%. The spectrum of the abaloe data set is domiated by the leadig 2 eigevalues, which accout for 99.97% of the spectrum (the first eigevalue aloe captures 99.3% of the spectrum). For this data set, the first 10 eigevalues capture all but % of the total spectrum. 5.2 Bechmark Partial Eigevalue Problem Algorithms. The proposed subspace trackig algorithm is compared with three other (batch) algorithms. The first two algorithms are the irbleigs algorithm of Baglama (2004)ad the laeig fuctio of the PROPACK fuctio set (Larse, 2004). These algorithms use Laczos methods to compute a partial eigevalue problem ad are assumed to require O(m 2 )operatios.toallowaobjective compariso of the computatioal time required to reach a solutio to the partial eigevalue problem, the Matlab implemetatio of these fuctios is used. The fial bechmark algorithm is the built-i Matlab fuctio eigs.thisfuctioisselectedoverthestadardeig or svd Matlab fuctios because it also uses a Laczos implemetatio, severely reducig computatioal requiremets. A straightforward, ooptimized script is used to implemet the trackig algorithm. All tests are performed i Matlab 7.2 usig a stadard computer equipped with a Itel Petium M GHz processor ad MB RAM.

18 Trackig the Domiat Normalized Kerel Eigespace 539 Figure 2: The first 50 eigevalues of the eigespectra of the ormalized kerel matrices for the artificial data set (A) ad the abaloedata set (B). 5.3 Performace Measure. The measure of accuracy betwee two matrices cosidered i this sectio is the relative error: δ L = L L F L F = Ũ Ũ T L F L F. (5.2)

19 540 G. Gis, I. Smets, ad J. Va Impe Here, F deotes the Frobeius orm, L is the approximatio of the ormalized kerel matrix, ad Ũ ad are, respectively, eigevector ad eigevalue matrices computed by meas of batch computatios or the trackig algorithm. Likewise, the relative error is used to express the differece betwee the estimated eigevalue σ ad σ,theeigevalueobtaiedthroughbatch computatios: δ σ σ σ =. σ (5.3) 5.4 Sigle Poit Updatig Performace. The sigle poit updatig performace of the algorithm is studied by addig a radom ew data poit to a radom base set of traiig samples ad computig the domiat eigespace of dimesio ( + 1) m. Aftereachupdatestep, L U, the approximatio obtaied usig the algorithm proposed i sectio 3.1 ad L B,thebestrak-m approximatio of the real Laplacia matrix obtaied from batch SVD calculatios, is compared with the full-rak matrix L. I additio, the eigevalues computed by the updatig algorithm are compared to those of the batch computatios,obtaied usig irbleigs, laeig, ad eigs. This procedure was repeated 2000 times. I all update steps, the orthogoality of the eigevectors was preserved; o reorthogoalizatio was ecessary. The average relative errors δ L betwee the approximated Laplacias L U ad L B, ad the full-rak matrix L, are listed i Table 2 for several (m, ) combiatios,togetherwiththeirstadarddeviatios.becausethe laeig ad eigs algorithms yield early idetical results, oly the eigs results are tabulated. From these results, it is clear that the performace of the updatig scheme is more tha satisfactory, with relative errors uder 1% for the artificial data set ad uder 0.01% for the abaloe data set. The accuracy icreases with icreasig data set size ad icreasig eigespace dimesio m. However, where the batch SVD computatios cotiue to yield better approximatios as m icreases, the updatig scheme has a maximum accuracy. For the batch computatios, the irbleigs algorithm is cosistetly outperformed by the laeig ad eigs algorithms. The average relative errors δ σ for the largest 10 eigevalues are depicted i Figure 3 for m = 10 (filled circles) ad m = 20 (ope circles), for a data set size of 500. Larger data set sizes produce similar plots with eve lower relative errors. From these plots, it ca be see that the updatig scheme yields very good approximatios of the real eigevalues, obtaied by batch computatios: the average relative errors are all below 0.01% for both data sets, ad most exhibit a average relative error of approximately 0.001%. Also, trackig

20 Trackig the Domiat Normalized Kerel Eigespace 541 Table 2: Compariso of the Sigle Poit Updatig Performace of the Proposed Updatig Algorithm ad Batch SVD Computatios. m δl Updatig Algorithm δl irbleigs δl eigs Artificial data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Abaloe data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±

21 542 G. Gis, I. Smets, ad J. Va Impe Figure 3: Absolute value of the average relative errors δ σ o the leadig 10 eigevalues for the artificial data set (A) ad the abaloe data set (B) for m = 10 (filled circles) ad m = 20 (ope circles) durig sigle poit updatig. The data set size is 500. more eigevalues tha strictly eeded icreases the accuracy for the smaller eigevalues, as evideced clearly i the abaloe data set. Although the updatig algorithm does ot exhibit the pipoit accuracy of the batch SVD calculatios, it geerally exhibits very small relative errors δ L.Iadditio,thedomiateigevaluesareverywellapproximated,as

22 Trackig the Domiat Normalized Kerel Eigespace 543 evideced by the small values for δ σ. Hece, it ca be cocluded that the updatig algorithm is performig well for sigle poit updatig i all test cases, providig excellet results i oly a fractio of the time required to perform batch computatios. 5.5 Trackig Performace. Next, the performace of the trackig algorithm itself is ivestigated. Here, a m eigespace is tracked durig 3000 iteratio steps. As with the sigle poit updatig, the eigespaces obtaied usig the trackig algorithm ad batch SVD computatios are compared with the full-rak Laplacia L after each iteratio step. This procedure was repeated 20 times. No reorthogoalizatios were required i ay of the iteratio steps. Table 3 reflects the average values for the relative errors δ L over all iteratio steps. As was observed with sigle poit updatig i sectio 5.4, the trackig algorithm caot match the batch computatios pipoit accuracy. However, the performace of the trackig scheme is still more tha satisfactory, with relative errors δ L uder 5% for the artificial data set ad uder 0.1% for the abaloe data set. As was observed with sigle poit updatig, icreasig the dimesio of the eigespace m iitially improves trackig performace, util a maximal accuracy is reached. However, if too may eigevalues are tracked, a small loss of accuracy is observed. Figure 4 depicts the average relative error δ σ for the 10 leadig eigevalues, for m = 10 (filled circles) ad m = 20 (ope circles), agai for a data set size of 500. As i sectio 5.4, icreasig the data set size yields similar plots, with slightly lower relative errors δ σ. It is clear that the eigevalues are tracked closely, with relative errors of approximately 1% for the artificial data set. For the abaloe data set, the figure also demostrates that while larger eigevalues are tracked accurately (δ σ 0.1%), the trackig error icreases as the magitude of the eigevalues decreases. However, give the egligible cotributio of these smaller eigevalues to the total spectrum, 7 these errors are still acceptable: i practical applicatios, oly the first two eigevalues ad eigevectors of the abaloe data set would be tracked. It is also importat to moitor the evolutio of these relative errors. If these errors accumulate over each iteratio step, the trackig algorithm will evetually become ustable. The evolutio of the relative error δ L is illustrated i Figure 5, ad the evolutio of δ σ for the first two eigevalues is depicted i Figure 6. A importat coclusio is that the relative errors δ L ad δ σ remai statioary over all iteratio steps. This is a valuable result, as it implies that the updatig scheme is a appropriate tool for trackig the eigespace durig a large umber of iteratio steps before retraiig (e.g., by meas of a batch computatio) is required. 7 Eigevalues 3 to 10 represet 0.02% of the total spectrum.

23 544 G. Gis, I. Smets, ad J. Va Impe Table 3: Compariso of the Trackig Performace of the Proposed Trackig Algorithm ad Batch SVD Computatios. m δl Trackig Algorithm δl irbleigs δl eigs Artificial data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Abaloe data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±

24 Trackig the Domiat Normalized Kerel Eigespace 545 Figure 4: Average relative errors δ σ o the leadig 10 eigevalues for the artificial data set (A) ad the abaloe data set (B) for m = 10 (filled circles) ad m = 20 (ope circles) durig 3000 trackig iteratios. The data set size is 500. Based o these results, it ca be cocluded that the trackig algorithm exhibits good to excellet trackig behavior. 5.6 Multiple Poit Updatig Performace. Whe the domiat eigespace of a very large data set is eeded or whe the available data set grows i size over time, it is ot always possible to perform batch

25 546 G. Gis, I. Smets, ad J. Va Impe Figure 5: Evolutio of the relative errors δ L o the Laplacia L durig 3000 trackig iteratios for the artificial data set (A) ad the abaloe data set (B), for adatasetsize of 500 ad a eigespace dimesio size m of 10. computatios because of time or computer memory limitatios. Therefore, the covergece of the updatig algorithm to the real eigespace of a large data set is tested. Startig from a iitial eigespace of dimesio m, ew data poits are added, updatig the eigespace i each step. After addig 2000 data poits, the quality of the approximatio is ivestigated. To study the evolutio of the approximatio quality whe addig more data poits to the base set, itermediate results are computed after addig 500

26 Trackig the Domiat Normalized Kerel Eigespace 547 Figure 6: Evolutio of the relative errors δ σ o the first two eigevalues durig 3000 trackig iteratios for the artificial data set (A) ad the abaloe data set (B). The data set size is 500, ad the eigespace dimesio size m is 10. data poits. This procedure was repeated 20 times. No reorthogoalizatios were required. Table 4 lists the itermediate approximatio results, ad the results for the full data set are summarized i Table 5. As observed i sectios 5.4 ad 5.5, the updatig scheme agai caot match the precisio of batch computatios but evertheless exhibits a acceptable precisio.

27 548 G. Gis, I. Smets, ad J. Va Impe Table 4: Itermediate Approximatio Accuracy for the Domiat Eigespace of Dimesio m of a Large Data Set After Addig 500 Data Poits to a Data Set of Iitial Size. m δl Updatig Algorithm δl irbleigs δl eigs Artificial data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Abaloe data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±

28 Trackig the Domiat Normalized Kerel Eigespace 549 Table 5: Fial Approximatio Accuracy for the Domiat Eigespace of Dimesio m of a Large Data Set After Addig 2000 Data Poits to a Data Set of Iitial Size. m δl Updatig Algorithm δl irbleigs δl eigs Artificial data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Abaloe data set ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±

29 550 G. Gis, I. Smets, ad J. Va Impe Figure 7: Fial relative errors δ σ o the leadig 10 eigevalues for the artificial data set (A) ad the abaloe data set (B) for m = 10 (filled circles) ad m = 20 (ope circles) durig multiple poit updatig. The iitial data set size is 500. Whe comparig the itermediate results of Table 4 to the fial results of Table 5, it is clear that the approximatio errors δ L i both tables are comparable. Thus, the approximatio errors remai stable while addig ew data poits, idicatig a good scalability of the method. Figure 7 depicts the fial relative errors δ σ for the largest 10 eigevalues, after addig 2000 data poits to a iitial data set of size = 500, for m = 10

30 Trackig the Domiat Normalized Kerel Eigespace 551 (filled circles) ad m = 20 (ope circles). Agai, icreasig the iitial data set results i similar plots, with slightly lower relative errors. It is clear that the domiat eigevalues are very well approximated, with relative errors δ σ uder 1% for the artificial data set. For the abaloe data set, results similar to those of sectio 5.5 are observed: the leadig two eigevalues are very well approximated (δ σ 0.003%), while the relative errors o subsequet eigevalues icrease sigificatly. Agai, these larger errors are still acceptable give the very small cotributio of these eigevalues to the power spectrum of the full data set. As evideced by these results, the updatig part of the trackig algorithm is well suited to approximate the domiat eigespace of the Laplacia kerel matrix of a large data set that grows i size over time. While the algorithm caot compete with batch SVD computatios for pure accuracy, it provides a valid ad rapid alterative. Whe oly the eigespace decompositio of the fial data set is required, the updatig algorithm will compute subsequet approximatios of the domiat eigespace, requirig a great umber of iteratio steps to reach to fial result. I cotrast, the batch computatios will immediately compute the fial decompositio, which results i a lower computatio time. Therefore, batch Laczos computatios are the superior choice i this situatio. 5.7 Computatioal Aspects. Fially, the computatioal requiremets of the proposed updatig ad trackig algorithms are compared with those of the batch algorithms. For various combiatios of data set size ad eigespace dimesio m, thetimerequiredbythevariousalgorithmsto compute a sigle trackig step is measured usig the built-i Matlab timer fuctio. Figure 8 compares the results for varyig data set size.ascabesee, the proposed trackig algorithm is sigificatly faster tha the tested batch algorithms, obtaiig its results approximately twice as fast as the irbleigs fuctio ad the built-i Matlab fuctio eigs. The laeig fuctio of the PROPACK fuctio takes more tha three times loger tha the proposed trackig algorithm to compute a update of the domiat eigespace decompositio. A importat observatio is that while all algorithms claim a O(m 2 )liearcomplexityithedatasetsize,theobservedcomputatioal complexity is slightly higher i this test. Figure 9 displays the required computatio times for varyig values of the eigespace dimesio m. Agai, the proposed trackig algorithm is sigificatly faster for all tested values of m ad exhibits excellet scalig properties. A icrease i the tracked eigespace dimesio from 2 to 40 results i a icrease i computatio time of 47% for the proposed trackig algorithm. I compariso, the same icrease i eigespace dimesio leads to a icrease i computatio time of 210%, 382%, ad 250% for, respectively, the irbleigs, laeig, adeigs algorithms.

31 552 G. Gis, I. Smets, ad J. Va Impe Figure 8: Computatio time required to perform a sigle trackig iteratio usig the proposed trackig algorithm (filled circles), irbleigs (ope circles), laeig (plus sigs), ad eigs ( ). The eigespace dimesio m is 10. Figure 9: Computatio time required to perform a sigle trackig iteratio usig the proposed trackig algorithm (filled circles), irbleigs (ope circles), laeig (plus sigs), ad eigs ( ). The data set size is 1000.

32 Trackig the Domiat Normalized Kerel Eigespace 553 These test results idicate that the proposed trackig algorithm is capable of quickly ad efficietly computig the domiat eigespace of a ormalized kerel matrix. A optimized implemetatio of the trackig algorithm, istead of a straightforward, ooptimized oe, might further improve the results. 6 Coclusio Kerel-based methods, frequetly exploited i classificatio, patter recogitio, fuctio estimatio, system idetificatio, ad sigal processig problems, are powerful tools because they are capable of solvig highly oliear problems by reformulatig them i a liear cotext. Here, the domiat eigespace of a (ormalized) kerel matrix is ofte a prerequisite. Ufortuately, the computatioal requiremets of the curret methods restrict their applicability to relatively small data sets. Recetly a algorithm able to efficietly track the domiat eigespace of a oormalized kerel matrix was proposed by Hoegaerts et al. (2004, 2007). Although it performs very well, it is ot able to track the domiat eigespace of ormalized kerel matrices, which geerally yield better results tha oormalized kerel matrices (Weiss, 1999). This letter therefore proposes a trackig algorithm for the domiat eigespace of a ormalized kerel matrix of a large data set by combiig two umerically stable subalgorithms. The first is capable of computig the update after the additio of a ew data poit, iducig a extesio of the Laplacia matrix i both its row ad colum dimesios while all matrix elemets are rescaled. The secod subalgorithm performs dowdatig: the exclusio of a existig data poit from the data set. Whe tested i some represetative case studies (based o a artificial ad the abaloe data set) ad compared to three batch SVD computatio algorithms, it is clear that a quite satisfactory approximatio of the domiat eigespace ca be obtaied usig the trackig algorithm proposed i this letter. The loss i accuracy with respect to batch SVD calculatios is by far compesated by the reduced computatioal ad memory requiremets per iteratio step, beig O(m 2 ) ad O(m), respectively. Ackowledgmets This work was supported i part by projects CoE EF/05/006 Optimizatio i Egieerig (OPTEC) ad OT/03/30 of the Research Coucil of the Katholieke Uiversiteit Leuve, ad the Belgia Program o Iteruiversity Poles of Attractio, iitiated by the Belgia Federal Sciece Policy Office. The scietific resposibility is assumed by its authors.

33 554 G. Gis, I. Smets, ad J. Va Impe Refereces Baglama, J. (2004). The irbleigs Matlab program for computig a few eigevalues ad eigevectors of a large sparse hermitia matrix. Available olie at jbaglama/. Blake, C., & Merz, C. (1998). UCI repository of machie learig databases. Available olie at mlear/mlrepository.html. Golub, G. H., & Va Loa, C. F. (1996). Matrix computatios. Baltimore,MD:Joh Hopkis Uiversity Press. Hoegaerts, L., De Lathauwer, L., Suykes, J. A. K., & Vadewalle, J. (2004). Efficietly updatig ad trackig the domiat kerel eigespace. I Proceedigs of the 16th Iteratioal Symposium o Mathematical Theory of Networks ad Systems (MTNS 2004). Leuve,Belgium. Hoegaerts, L., De Lathauwer, L., Suykes, J. A. K., & Vadewalle, J. (2007). Efficietly updatig ad trackig the domiat kerel pricipal compoets. Neural Networks, 20(2), Kim, K., Fraz, M. O., & Schölkopf, B. (2003). Kerel Hebbia algorithm for iterative kerel pricipal compoet aalysis (Tech. Rep. 109). Tübige. Max-Plack- Istitut für biologische Kyberetik. Larse, R. M. (2004). Propack software for large ad sparse SVD calculatios. Available olie at rmuk/propack/idex.html. Ng, A. Y., Jorda, M. I., & Weiss, Y. (2002). O spectral clusterig: Aalysis ad a algorithm. I S. Becker, T. G. Dietterich, & Z. Ghahramai (Eds.), Advaces i eural iformatio processig systems. Cambridge, MA: MIT Press. Rosipal, R., & Girolami, M. (2001). A expectatio-maximizatio approach to oliear compoet aalysis. Neural Computatio, 13(3), Schölkopf, B., & Smola, A. (2002). Learig with kerels. Cambridge, MA: MIT Press. Schölkopf, B., Smola, A., & Müller, K. (1998). Noliear compoet aalysis as a kerel eigevalue problem. Neural Computatio, 10, Suykes, J. A. K., Va Gestel, T., De Brabater, J., De Moor, B., & Vadewalle, J. (2002). Least squares support vector machies. Sigapore: World Scietific. Weiss, Y. (1999). Segmetatio usig eigevectors: A uifyig view. I Proceedigs IEEE Iteratioal Coferece o Computer Visio (Vol. 9, pp ). Piscataway, NJ: IEEE Press. Received May 10, 2006; accepted February 13, 2007.