Diffusion Kernels on Graphs and Other Discrete Structures

Transcription

1 Diffusion Kernels on Graphs and Other Disrete Strutures Risi Imre Kondor ohn Lafferty Shool of Computer Siene Carnegie Mellon University Pittsburgh P 523 US KONDOR@CMUEDU LFFERTY@CSCMUEDU bstrat The appliation of kernel-based learning algorithms has so far largely been onfined to realvalued data and a few speial data types suh as strings In this paper we propose a general method of onstruting natural families of kernels over disrete strutures based on the matrix exponentiation idea In partiular we fous on generating kernels on graphs for whih we propose a speial lass of exponential kernels based on the heat equation alled diffusion kernels and show that these an be regarded as the disretization of the familiar Gaussian kernel of Eulidean spae Introdution Kernel-based algorithms suh as Gaussian proesses Makay 7 support vetor mahines urges and kernel PC Mika et al are enjoying great popularity in the statistial learning ommunity The ommon idea behind these methods is to express our prior beliefs about the orrelations or more generally the similarities between pairs of points in data spae in terms of a kernel funtion and thereby to impliitly onstrut a mapping to a Hilbert spae in whih the kernel appears as the inner produt!"$%"&!' Shölkopf & Smola 200 With respet to a basis of eah datapoint then splits into a possibly infinite number of independent features a property whih an be exploited to great effet Graph-like strutures our in data in many guises and in order to apply mahine learning tehniques to suh disrete data it is desirable to use a kernel to apture the longrange relationships between data points indued by the loal struture of the graph One obvious example of suh data is a graph of douments related to one another by links suh as the hyperlink struture of the World Wide Web Other examples inlude soial networks itations between sientifi artiles and networks in linguistis lbert & arabási 200 Graphs are also sometimes used to model ompliated or only partially understood strutures in a first approximation In hemistry or moleular biology for example it might be antiipated that moleules with similar hemial strutures will have broadly similar properties While for two arbitrary moleules it might be very diffiult to quantify exatly how similar they are it is not so diffiult to propose rules for when two moleules an be onsidered neighbors for example when they only differ in the presene or absene of a single funtional group movement of a bond to a neigbouring atom et Representing suh relationships by edges gives rise to a graph eah vertex orresponding to one of our original objets Finally adjaeny graphs are sometimes used when data is expeted to be onfined to a manifold of lower dimensionality than the original spae Saul & Roweis 200 elkin & Niyogi 200 and Szummer & aakkola 2002 In all of these ases the hallenge is to apture in the kernel the loal and global struture of the graph In addition to adequately expressing the known or hypothesized struture of the data spae the funtion must satisfy two mathematial requirements to be able to serve as a kernel: it must be symmetri * + and positive semi-definite Construting appropriate positive definite kernels is not a simple task and this has largely been the reason why with a few exeptions kernel methods have mostly been onfined to Eulidean spaes where several families of provably positive semi-definite and easily interpretable kernels are known When dealing with intrinsially disrete data spaes the usual approah has been either to map the data to Eulidean spae first as is ommonly done in text lassifiation treating integer word ounts as real numbers oahims or when no suh simple mapping is forthoming to forgo using kernel methods altogether notable exeption to this is the line of work stemming from the onvolution kernel idea introdued in Haussler and related but independently oneived ideas on string kernels first presented in Watkins Despite the promise of these ideas relatively little work has been done on disrete kernels sine the publiation of these artiles In this paper we use ideas from spetral graph theory to propose a natural lass of kernels on graphs whih we refer to as diffusion kernels We start out by presenting a more general lass of kernels alled exponential kernels appli-

2 < N N E able to a wide variety of disrete objets In Setion 3 we present the ideas behind diffusion kernels and the interpretation of these kernels on graphs In Setion 4 we show how diffusion kernels an be omputed for some speial families of graphs and these tehniques are further developed in Setion 5 Experiments using diffusion kernels for lassifiation of ategorial data are presented in Setion 6 and we onlude and summarize our results in Setion 7 2 Exponential kernels In this setion we show how the exponentiation operation on matries naturally yields the ruial positive-definite riterion of kernels and desribe how to build kernels on the diret produt of graphs Reall that in the disrete ase positive semi-definiteness amounts to for all sets of real oeffiients ase 2 and in the ontinuous + for all square integrable real funtions the latter is sometimes referred to as Merer s ondition In the disrete ase for finite represented by an the kernel an be uniquely matrix whih we shall denote by the same letter with rows and olumns indexed by the elements of and related to the kernel by Sine this matrix alled the Gram matrix and the funtion are essentially equivalent in partiular the matrix inherits the properties of symmetry and positive semi-definiteness we an refer to one or the other as the kernel without risk of onfusion The exponential of a square matrix is defined as "! $&%' *+-0/2 354 where the limit always exists and is equivalent to 6! 7 / / : / =< >?: 3 4 It is well known that any even power of a symmetri matrix is positive semi-definite and that the set of positive semi-definite matries is omplete with respet to limits of sequenes under the Frobenius norm Taking to be symmetri and replaing 3 by 3 shows that the exponential of any symmetri matrix is symmetri and positive semidefinite hene it is a andidate for a kernel Conversely it is easy to show that any infinitely divisible kernel an be expressed in the exponential form 3 Infinite divisibility means that an be written as an 3 -fold onvolution Haussler Suh kernels form on- for any 3 tinuous families G E FE DC indexed by a real parameter and are related to infinitely divisible probability distributions whih are the limits of sums of independent random variables 6K Feller 7 The tautology beomes as 3 goes to infinity IH C L%&' *M+ 0/ 3 6PQ 4 ON whih is equivalent to 3 for SR R 6PQ The above already suggests looking for kernels over finite sets in the form 6! 5 guaranteeing positive definiteness without seriously restriting our hoie of kernel Furthermore differentiating UT with respet to and examining the resulting differential equation V 6 with the aompanying initial onditions W 27 lends itself natually to the interpretation that is the produt of a ontinuous proess expressed by gradually transforming it from the identity matrix W to a kernel with stronger and stronger off-diagonal effets as inreases We shall see in the examples below that by virtue of this relationship hoosing to express the loal struture of will result in the global struture of naturally emerging in We all XT an exponential family of kernels with generator and bandwidth parameter Note that the exponential kernel onstrution is not related to the result desribed in erg et al 4 Haussler and Shölkopf & Smola 200 based on Shoenberg s pioneering work in the late 30 s in the theory of positive definite funtions Shoenberg 3 This work shows that any positive semi-definite an be written as ZY\[ ^ `_ where a is a onditionally positive semi-definite kernel that is it satisfies 2 under the additional onstraint that bd V Whereas 5 involves matrix exponentiation via 3 formula 7 presribes the more straight-forward omponentwise exponentiation On the other hand onditionally Instead of using the term onditionally positive definite this type of objet is sometimes referred to by saying that e0f is negative definite Confusingly a negative definite kernel is then not the same as the negative of a positive definite kernel so we shall avoid using this terminology 7

3 7! 7 b = positive definite matries are somewhat elusive mathematial objets and it is not lear where Shoenberg s beautiful result will find appliation in statistial learning theory The advantage of our relatively brute-fore approah to onstruting positive definite objets is that it only requires that the generator be symmetri more generally self-adjoint and guarantees the positive semi-definiteness of the resulting kernel There is a anonial way of building exponential kernels over diret produts of sets whih will prove invaluable in what follows Let be a family of kernels over the and let be a family of set with generator kernels over with generator To onstrut an exponential kernels over the pairs with and it is natural to use the generator H 2H / H where if and otherwise In other words we take the generator over the produt set to be / where 7 and 7 are the and dimensional diagonal kernels respetively Plugging into 6 shows that the orresponding kernels will be given simply by H H H that is any exponential kernel on to an exponential kernel over length by P In partiular we an lift or using the tensor produt notation 3 Diffusion kernels on graphs n undireted unweighted graph is defined by a vertex set and an edge set the latter being the set of unordered pairs where whenever the verties and are joined by an edge denoted Equation 6 suggests using an exponential kernel with generator for " % $ for "" otherwise where is the degree of vertex " number of edges emanating from vertex " The negative of this matrix sometimes up to normalization is alled the Laplaian of and it plays a entral role in spetral graph theory Chung 7 It is instrutive to note that for any vetor & ' *' &+ -& % / & 0 % & P showing that is in fat negative semi-definite ting on funtions 2 by W an also be regarded as an operator In fat it is easy to show that on a square grid in 3 - dimensional Eulidean spae with grid spaing is just the finite differene approximation to the familiar ontinuous Laplaian : / and that in the limit 4 this approximation beomes exat In analogy with lassial physis where equations of the form are used to desribe the diffusion of heat and other substanes through ontinuous media our equation 0 with as defined in is alled the heat equation on and the resulting kernels are alled diffusion or heat kernels 3 stohasti and a physial model There is a natural lass of stohasti proesses on graphs whose ovariane struture yields diffusion kernels Consider the random field obtained by attahing independent zero mean variane random variables W to eah vertex " Now let eah of these random variables send a fration CED of their value to eah of their respetive neighbors at disrete that is let /V 2 / C GF H I Introduing the time evolution operator I $ / C L ' M' % The ovariane of the random field at time N Cov OVH P VHN SRUT whih simplifies to Cov N % Q [ + an be written as X VWT P X [ [ E is % Q W YV<Z 2

4 K & [ Q / by independene at time zero H that the is "% Note holds regardless of the partiular distribution of X as long as their mean is zero and their variane by repla- Now we an derease the time step from to ing by 5 and C by C in 2 giving 0/ C 5 4 K C whih in the limit is exatly of the form 3 In partiular the ovariane beomes the diffusion kernel Cov - K! Sine kernels are in some sense nothing but generalized ovarianes in fat in the ase of Gaussian Proesses they are the ovariane this example supports the ontention that diffusion kernels are a natural hoie on graphs Closely related to the above is an eletrial model Differentiating with respet to yields the differential equations C GF H< P % These equations are the same as those desribing the relaxation of a network of apaitors of unit apaitane where one plate of eah apaitor is grounded and the other plates are onneted aording to the graph struture eah edge orresponding to a onnetion of resistane 5 C The then measure the potential at eah apaitor at time In partiular " C is the potential at apaitor " time after having initialized the system by deharging every apaitor exept for apaitor whih starts out at unit potential 32 The ontinuous limit s a speial ase it is instrutive to again onsider the in- finitely fine square grid on Introduing the similarity funtion the heat equation 0 gives & Sine the Laplaian is a loal operator in the sense that in the neigh- is only affeted by the behavior of borhood of as long as is ontinuous in the above an be rewritten as It is easy to verify that the solution of this equation with Dira spike initial onditions Gaussian % is just the 2 Note that is here used to denote infinitesimals and not the Laplaian & ' ' C [ _ showing that similarity to any given point as expressed by the kernel really does behave as some substane diffusing in spae and also that the familiar Gaussian kernel on & ' ' C [ is just a diffusion kernel with 2 5 In this sense diffusion kernels an be regarded as a generalization of Gaussian kernels to graphs 33 Relationship to random walks It is well known that diffusion is losely related to random walks random walk on an unweighted graph is a stohasti proess generating where in suh a way that S" 5Z if " and zero otherwise $ lazy random walk on with parameter 5 ' is very similar exept that when at vertex " the proess will take eah of the edges emanating from " with fixed probability ie! " 2" for " will remain in plae with probability % Considering the distribution! $ % Q in the limit with 5 and leads exatly to 3 showing that diffusion kernels are the ontinuous time limit of lazy random walks This analogy also shows that "% an be regarded as a sum over paths from " to namely the sum of the probabilities that the lazy walk takes eah path For graphs in whih every vertex is of the same degree mapping eah vertex " to every path starting at " weighted by the square root of the probability of a lazy random walk starting at " taking that path ' ' ' " R C F+*- P : 4 ' ' 0 _ C where is the set of all paths on gives a representation of the kernel in the spae 23 of linear ombinations of paths of the form 6 54 [ _ for loops ie Q ' ' otherwise where basis of loops 5 7 ' ' @ Q is the reverse of In the : Q <: and linear ombinations [ %

5 Q Q! $ 3 for all pairs of non-loops this does give a diagonal representation of but not a representation satisfying beause there are alternating /F s and % s on the diagonal 34 Diffusion on weighted graphs Finally we remark that diffusion kernels are not restrited to simple unweighted graphs For multigraphs or weighted N symmetri graphs all we need to do is to set " to be the total weight of all edges between " and and reweight the diagonal terms aordingly The rest of the analysis arries through as before 4 Some speial graphs In general omputing exponential kernels involves diagonalizing the generator whih is always possible beause omputing is symmetri and then "! whih is easy beause will be diagonal with H & & The diagonalization proess is omputationally expensive however and the kernel matrix must be stored in memory during the whole time the learning algorithm operates Hene there is interest in the few speial ases for whih the kernel matrix an be omputed diretly 4 -regular trees n infinite -regular tree is an undireted unweighted graph with no yles in whih every vertex has exatly neighbors Note that this differs from the notion of a rooted 3 -ary tree in that no speial node is designated the root ny vertex an funtion as the root of the tree but that too must have exatly neighbors Hene a 3-regular tree looks muh like a rooted binary tree exept that at the root it splits into three branhes and not two Sine in this graph every vertex is reated perfetly equal "% an only depend on the relative positions of " and namely the length of the unique path between them &" Chung and Yau show that " [ [ _ R ^ "% 3 / % % + % % H + % D%W / % % X % for V "% and "%" [ [ _ R X 4 + % % % D% % / %!" for the diagonal elements 42 Complete graphs In the unweighted omplete graph with 3 verties any pair of verties is joined by an edge hene "% 3 It is easy to verify that the orresponding solution to 0 is "% % 0/ 3 % 3 for " % 3 for " 5 showing that with inreasing the kernel relaxes exponentially to "% 5 3 The asymptotially exponential harater of this solution and the onvergene to the uniform kernel for finite are diret onsequenes of the fat that is a linear operator and we shall see this type of behavior reur in other examples 43 Closed hains When is a single losed hain of length 3 "% will learly only depend on the distane &" along the hain between " and Labeling the verties onseutively from to 3 % the similarity funtion at a partiular vertex without loss of generality vertex zero an be expressed in terms of its disrete Fourier transform & X Q 3 The heat equation implies Q Q / ' 3 % $ PQ % Q Q / Q whih after some trigonometry translates into % & Q % % " & 3 4 % ' 3 $ % & Q showing that the Fourier oeffiients deay independently of one another Using the inverse Fourier transform the solution orresponding to the initial ondition Q " at will be Q 3 & +*- $ PQ where $ 0/ % " "% 3 & 3* " % $ P Q $ "% 2 and the kernel itself will be

6 > *! [ _ < [ _ < $ * 44 The hyperube and tensor produts of omplete graphs Kernels on the speial graphs onsidered above an serve as building bloks for tensor produt kernels as in For example it is natural to identify binary strings * with the of @ of the 3 -dimensional hyperube Construting a diffusion kernel on the hyperube regarded as a graph amounts to asserting that two sequenes are neighbors if they only differ in a single digit From 5 and the diffusion kernel on the hyperube will be % R [ _ 0/ 4 R [ _ whih only depends on the Hamming distane between and and is extremely easy to ompute Similarly the diffusion kernel on strings over an alphabet size \ will be % ' ' 0/ % '>' 4 R [ _ where is the number of harater plaes at whih and differ 5 Conjugay the method of images and string kernels Conjugating the Gram matrix by a not neessarily square matrix [ _ yields a new positive semi-definite kernel of the form [ _ One appliation of this is in reating virtual data points We have noted above that the distintion between -regular trees and infinite + % -ary rooted trees is that arbitrarily designating a vertex in the former as the root we find that it has an extra branh emanating from it Figure Taking for simpliity the analytial formulæ 3 and 4 are hene not diretly appliable to binary rooted trees beause if we simply try to ignore this branh by not mapping any data points to it in the language of the eletrial analogy of Setion 3 we find that some urrent will seep away through it The ruial observation is that the graph possesses mirror symmetry about the edge onneting this errant branh to the rest of the graph Mapping eah vertex " of the binary + of Figure The three-regular tree left whih extends to infinity in all diretions little bending of the branhes shows that it is isomorphi to two rooted binary trees joined at the root right The method of images enables us to ompute the diffusion kernel between verties of the binary tree by mapping eah to a pair of verties and on the three-regular tree and summing the ontributions from and 6 tree to the analogous vertex on one side of this plane of symmetry ' " in the > -regular tree and its mirror image ' " on the other side solves the problem beause by symmetry in the eletrial analogy the flow of urrent aross the ritial edge onneting the two halves of the graph will be zero This onstrution alled the method of images orresponds to a transformation matrix of the form and yields in analytial form the diffusion kernel for infinite binary trees "% R X&" / p R " q X&" &% / &'% / 6 where % designates the root and measures distanes on the binary tree nother appliation of onjugated diffusion kernels is in the onstrution of string kernels sensitive to mathing nonontiguous substrings The usual approah to this is to introdue blank haraters into the strings and to be ompared so that the haraters of the ommon substring are aligned Using the tensor produt of omplete graphs approah developed above it is easy to add an extra harater to the alphabet to represent We an then map and to a of dimensionality 3 / generalized hyperube by mapping eah string to the verties orresponding to all its posssible extensions by s Let us represent an alignment between and by the vetor of mathes +* * '* ' ' ' ' ' -

7 6 * and let be the set of all alignments between and ssuming that all virtual strings are weighted equally the resulting kernel will be ' ' 7 [ _ for some ombinatorial fator and % [ ' ' _ 0/ \ [ ' ' In the speial ase that 3 the ombinatorial fator beomes onstant for all pairs of strings and 7 beomes omputable by dynami programming by the reursion where / / if / otherwise For the derivation of reursive formulæ suh as this and omparison to other measures of similarity between strings see Durbin et al 6 Experiments on UCI datasets In this setion we desribe preliminary experiments with diffusion kernels fousing on the use of kernel-based methods for lassifying ategorial data For suh problems it is often quite unnatural to enode the data as vetors in Eulidean spae to allow the use of standard kernels However as our experiments show even simple diffusion kernels on the hyperube as desribed in Setion 44 an result in good performane for suh data For ease of experimentation we use a large margin lassifier based on the voted pereptron as desribed in Freund & Shapire 3 In eah set of experiments we ompare models trained using a diffusion kernel the Eulidean distane and a kernel based on the Hamming distane Data sets having a majority of ategorial variables were hosen any ontinuous features were ignored The diffusion kernels used are given by the natural extension of the hyperube kernels given in Setion 44 namely where P % '' / % ' ' 4 [ _ is the number of values in the alphabet of the " -th attribute Table shows sample results of experiments arried out on five UCI data sets having a majority of ategorial features 3 SVMs were trained on some of the data sets and the results were omparable to what we report here for the voted pereptron In eah experiment a voted pereptron was trained using 0 rounds for eah kernel Results are reported for the setting of the diffusion oeffiient ahieving the best rossvalidated error rate Sine the Eulidean kernel performed poorly in all ases the results for this lassifier are not shown The results are averaged aross 50 random splits of the training and test data In addition to the error rates also shown is the average number of support vetors or pereptrons used In general we see that the best lassifiers also have the sparsest representation The redution in error rate varies but the simple diffusion kernel generally performs well Figure 2 The average error rate left and number of support vetors right as a funtion of the diffusion oeffiient on the data set The horizontal line is the baseline performane using the Hamming kernel The performane over a range of values of on the! " data set is shown in Figure 2 We note that this is a very easy data set for a symboli learning algorithm sine it an be learned to an auray of about 50% with a few simple logial rules However standard kernels perform poorly on this data set and the Hamming kernel has an auray of only 664% The simple diffusion kernel brings the auray up to 23% 7 Conlusions We have presented a natural approah to onstruting kernels on graphs and related disrete objets by using the analogue on graphs of the heat equation on Riemannian manifolds The resulting kernels are easily shown to satisfy the ruial positive semi-definiteness riterion and they ome with intuitive interpretations in terms of random walks eletrial iruits and other aspets of spetral graph theory We showed how the expliit alulation of diffusion kernels is possible for speifi families of graphs and how the kernels orrespond to standard Gaussian kernels in a ontinuous limit Preliminary experiments on ategorial data where standard kernel methods were previously not appliable indiate that diffusion kernels an be effetively used with standard margin-based lassifiation shemes While the tensor produt onstrution allows one to inrementally build up more powerful kernels from simple omponents expliit formulas will be diffiult to ome by in general Yet the use of diffusion kernels may still be pratial when the underlying graph struture is sparse by using standard sparse matrix tehniques

8 Hamming Distane Diffusion Kernel Improvement error Data Set ttr ' \ error *\ error M\ " * 0 764% % % 3% 3 2 7% % % 5% 42 % 45 50% % % % % % 70% % 260 3% % 2% Table Results on five UCI data sets For eah data set only the ategorial features are used The olumn marked! indiates the maximum number of values for an attribute thus the data set has binary attributes Results are reported for the setting of the diffusion oeffiient ahieving the best error rate It is often said that the key to the suess of kernel-based algorithms is the impliit mapping from a data spae to some usually muh higher dimensional feature spae whih better aptures the struture inherent in the data The motivation behind the approah to building kernels presented in this paper is the realization that the kernel is a general representation of this inherent struture independent of how we represent individual data points Hene by onstruting a kernel diretly on whatever objet the data points naturally lie on eg a graph we an avoid the arduous proess of foring the data through any Eulidean spae altogether In effet the kernel trik is a method for unfolding strutures in Hilbert spae It an be used to unfold nontrivial orrelation strutures between points in Eulidean spae but it is equally valuable for unfolding other types of strutures whih intrinsially have nothing to do with linear spaes at all Referenes lbert R & arabási 200 Statistial mehanis of omplex networks vailable from "%$&$'&' ' *+ * $- '!$ / $ elkin M & Niyogi P 200 Laplaian eigenmaps for dimensionality redution and data representation Tehnial Report Computer Siene Department University of Chiago erg C Christensen & Ressel P 4 Harmoni analysis on semigroups: Theory of positive definite and related funtions Springer urges C C tutorial on support vetor mahines for pattern reognition Data Mining and Knowledge Disovery Chung F R K 7 Spetral graph theory No 2 in Regional Conferene Series in Mathematis merian Mathematial Soiety Chung F R K & Yau S-T Coverings heat kernels and spanning trees Eletroni ournal of Combinatoris 6 Durbin R Eddy S Krogh & Mithison G iologial sequene analysis probabilisti models of proteins and nulei aids Cambridge University Press Feller W 7 n introdution to probability theory and its appliations vol II Wiley Seond edition Freund Y & Shapire R E Large margin lassifiation using the pereptron algorithm Mahine Learning Haussler D Convolution kernels on disrete strutures Tehnial Report UCSC-CRL--0 Department of Computer Siene University of California at Santa Cruz oahims T Text ategorization with support vetor mahines: Learning with many relevant features Proeedings of ECML- 0th European Conferene on Mahine Learning pp Makay D C 7 Gaussian proesses: replaement for neural networks? NIPS tutorial vailable from "%$&$' &0 &23 4+% 5 6 &$ &7$ 4$ Mika S Shölkopf Smola Müller K Sholz M & Rätsh G Kernel PC and de-noising in feature spaes dvanes in Neural Information Proessing Systems Saul L K & Roweis S T 200 n introdution to loally linear embedding vailable from "%$&$'&' '%"6 &2 * $- ' $&00 &$ Shoenberg I 3 Metri spaes and ompletely monotone funtions The nnals of Mathematis 3 4 Shölkopf & Smola 200 Learning with kernels MIT Press Szummer M & aakkola T 2002 Partially labeled lassifiation with Markov random walks dvanes in Neural Information Proessing Systems Watkins C Dynami alignment kernels In Smola Shölkopf P artlett and D Shuurmans Eds dvanes in kernel methods MIT Press