arxiv: v4 [cs.si] 22 Dec 2017

Transcription

1 Graph Embeing Techniques, Applications, an Performance: A Survey Palash Goyal an Emilio Ferrara University of Southern California, Information Sciences Institute 4676 Amiralty Way, Suite 1. Marina el Rey, CA. 9292, USA arxiv:17281v4 [cs.si] 22 Dec 217 Abstract Graphs, such as social networks, wor co-occurrence networks, an communication networks, occur naturally in various real-worl applications. Analyzing them yiels insight into the structure of society, language, an ifferent patterns of communication. Many approaches have been propose to perform the analysis. Recently, methos which use the representation of graph noes in vector space have gaine traction from the research community. In this survey, we provie a comprehensive an structure analysis of various graph embeing techniques propose in the literature. We first introuce the embeing task an its challenges such as scalability, choice of imensionality, an features to be preserve, an their possible solutions. We then present three categories of approaches base on factorization methos, ranom walks, an eep learning, with examples of representative algorithms in each category an analysis of their performance on various tasks. We evaluate these state-of-the-art methos on a few common atasets an compare their performance against one another. Our analysis conclues by suggesting some potential applications an future irections. We finally present the open-source Python library we evelope, name GEM (Graph Embeing Methos, available at which provies all presente algorithms within a unifie interface to foster an facilitate research on the topic. Keywors: Graph embeing techniques, Graph embeing applications, Python Graph Embeing Methos GEM Library 1. Introuction Graph analysis has been attracting increasing attention in the recent years ue the ubiquity of networks in the real worl. Graphs (a.k.a. networks) have been use to enote information in various areas incluing biology (Protein-Protein interaction networks)[1], social sciences (frienship networks)[2] an linguistics (wor co-occurrence networks)[3]. Moeling the interactions between entities as graphs has enable researchers to unerstan the various network systems in a systematic manner[4]. For example, social networks have been use for applications like frienship or content recommenation, as well as for avertisement []. Graph analytic tasks can be broaly abstracte into the following four categories: (a) noe classification[6], (b) link preiction[], (c) clustering[7], an () visualization[8]. Noe classification aims at etermining the label of noes (a.k.a. vertices) base on other labele noes an the topology of the network. Link preiction refers to the task of preicting missing links or links that are likely to occur in the future. Clustering is use to fin subsets of similar noes an group them together; finally, visualization helps in proviing insights into the structure of the network. In the past few ecaes, many methos have been propose for the tasks efine above. For noe classification, there are broaly two categories of approaches methos which use ranom walks to propagate the labels[9, ], an methos which extract features from noes an apply classifiers on them[11, 12]. Approaches for link preiction inclue similarity base methos[13, 14], maximum likelihoo moels[1, 16], an probabilistic moels[17, 18]. Clustering methos inclue attribute base moels[19] an methos which irectly maximize (resp., minimize) the inter-cluster (resp., intra-cluster) istances[7, 2]. This survey will provie a taxonomy that captures these application omains an the existing strategies. Typically, a moel efine to solve graph-base problems either operates on the original graph ajacency matrix or on a erive vector space. Recently, the methos base on representing networks in vector space, while preserving their properties, have become wiely popular[21, 22, 23]. Obtaining such an embeing is useful in the tasks efine above. 1 The embeings are input as features to a moel an the parameters are learne base on the training ata. This obviates the nee for complex classification moels which are applie irectly on the graph Challenges Obtaining a vector representation of each noe of a graph is inherently ifficult an poses several challenges which have been riving research in this fiel: (i) Choice of property: A goo vector representation of noes shoul preserve the structure of the graph an the connection between iniviual noes. The first challenge is choosing 1 The term graph embeing has been use in the literature in two ways: to represent an entire graph in vector space, or to represent each iniviual noe in vector space. In this paper, we use the latter efinition since such representations can be use for tasks like noe classification, ifferently from the former representation. Preprint submitte to Knowlege Base Systems December 27, 217

2 the property of the graph which the embeing shoul preserve. Given the plethora of istance metrics an properties efine for graphs, this choice can be ifficult an the performance may epen on the application. (ii) Scalability: Most real networks are large an contain millions of noes an eges embeing methos shoul be scalable an able to process large graphs. Defining a scalable moel can be challenging especially when the moel is aime to preserve global properties of the network. (iii) Dimensionality of the embeing: Fining the optimal imensions of the representation can be har. For example, higher number of imensions may increase the reconstruction precision but will have high time an space complexity. The choice can also be application-specific epening on the approach: E.g., lower number of imensions may result in better link preiction accuracy if the chosen moel only captures local connections between noes Our contribution This survey provies a three-pronge contribution: (1) We propose a taxonomy of approaches to graph embeing, an explain their ifferences. We efine four ifferent tasks, i.e., application omains of graph embeing techniques. We illustrate the evolution of the topic, the challenges it faces, an future possible research irections. (2) We provie a etaile an systematic analysis of various graph embeing moels an iscuss their performance on the various tasks. For each metho, we analyze the properties preserve an its accuracy, through comprehensive comparative evaluation on a few common ata sets an application scenarios. Furthermore, we perform hyperparameter sensitivity analysis on the methos evaluate to test their robustness an provie an unerstaning of the epenence of optimal hyperparameter on the task performe. (3) To foster further research in this topic, we finally present GEM, the open-source Python library we evelope that provies, uner a unifie interface, implementations of all graph embeing methos iscusse in this survey. To the best of our knowlege, this is the first paper to survey graph embeing techniques an their applications Organization of the survey The survey is organize as follows. In Section 2, we provie the efinitions require to unerstan the problem an moels iscusse next. Section 3 proposes a taxonomy of graph embeing approaches an provies a escription of representative algorithms in each category. The list of applications for which researchers have use the representation learning approach for graphs is provie in Section 4. We then escribe our experimental setup (Section ) an evaluate the ifferent moels (Section 6). Section 7 introuces our Python library for graph embeing methos. Finally, in Section 8 we raw our conclusions an iscuss potential applications an future research irection Definitions an Preliminaries We represent the set {1,, n} by [n] in the rest of the paper. We start by formally efining several preliminaries which have been efine similar to Wang et al. [23]. Definition 1. (Graph) A graph G(V, E) is a collection of V = {v 1,, v n } vertices (a.k.a. noes) an E = {e i j } n i, j=1 eges. The ajacency matrix S of graph G contains non-negative weights associate with each ege: s i j. If v i an v j are not connecte to each other, then s i j =. For unirecte weighte graphs, s i j = s ji i, j [n]. The ege weight s i j is generally treate as a measure of similarity between the noes v i an v j. The higher the ege weight, the more similar the two noes are expecte to be. Definition 2. (First-orer proximity) Ege weights s i j are also calle first-orer proximities between noes v i an v j, since they are the first an foremost measures of similarity between two noes. We can similarly efine higher-orer proximities between noes. For instance, Definition 3. (Secon-orer proximity) The secon-orer proximity between a pair of noes escribes the proximity of the pair s neighborhoo structure. Let s i = [s i1,, s in ] enote the first-orer proximity between v i an other noes. Then, seconorer proximity between v i an v j is etermine by the similarity of s i an s j. Secon-orer proximity compares the neighborhoo of two noes an treats them as similar if they have a similar neighborhoo. It is possible to efine higher-orer proximities using other metrics, e.g. Katz Inex, Roote PageRank, Common Neighbors, Aamic Aar, etc. (for etaile efinitions, omitte here in the interest of space, see Ou et al. [24]). Next, we efine a graph embeing: Definition 4. (Graph embeing) Given a graph G = (V, E), a graph embeing is a mang f : v i y i R i [n] such that V an the function f preserves some proximity measure efine on graph G. An embeing therefore maps each noe to a low-imensional feature vector an tries to preserve the connection strengths between vertices. For instance, an embeing preserving firstorer proximity might be obtaine by minimizing i, j s i j y i y j 2 2. Let two noe pairs (v i, v j ) an (v i, v k ) be associate with connections strengths such that s i j > s ik. In this case, v i an v j will be mappe to points in the embeing space that will be closer each other than the mang of v i an v k. 3. Algorithmic Approaches: A Taxonomy In the past ecae, there has been a lot of research in the fiel of graph embeing, with a focus on esigning new embeing algorithms. More recently, researchers pushe forwar scalable embeing algorithms that can be applie on

3 graphs with millions of noes an eges. In the following, we provie historical context about the research progress in this omain ( 3.1), then propose a taxonomy of graph embeing techniques ( 3.2) covering (i) factorization methos ( 3.3), (ii) ranom walk techniques ( 3.4), (iii) eep learning ( 3.), an (iv) other miscellaneous strategies ( 3.6) Graph Embeing Research Context an Evolution In the early 2s, researchers evelope graph embeing algorithms as part of imensionality reuction techniques. They woul construct a similarity graph for a set of n D-imensional points base on neighborhoo an then embe the noes of the graph in a -imensional vector space, where D. The iea for embeing was to keep connecte noes closer to each other in the vector space. Laplacian Eigenmaps [2] an Locally Linear Embeing (LLE) [26] are examples of algorithms base on this rationale. However, scalability is a major issue in this approach, whose time complexity is O( V 2 ). Since 2, research on graph embeing has shifte to obtaining scalable graph embeing techniques which leverage the sparsity of real-worl networks. For example, Graph Factorization [21] uses an approximate factorization of the ajacency matrix as the embeing. LINE [22] extens this approach an attempts to preserve both first orer an secon proximities. HOPE [24] extens LINE to attempt preserve highorer proximity by ecomposing the similarity matrix rather than ajacency matrix using a generalize Singular Value Decomposition (SVD). SDNE [23] uses autoencoers to embe graph noes an capture highly non-linear epenencies. The new scalable approaches have a time complexity of O( E ) A Taxonomy of Graph Embeing Methos We propose a taxonomy of embeing approaches. We categorize the embeing methos into three broa categories: (1) Factorization base, (2) Ranom Walk base, an (3) Deep Learning base. Below we explain the characteristics of each of these categories an provie a summary of a few representative approaches for each category (cf. Table 1), using the notation presente in Table Factorization base Methos Factorization base algorithms represent the connections between noes in the form of a matrix an factorize this matrix to obtain the embeing. The matrices use to represent the connections inclue noe ajacency matrix, Laplacian matrix, noe transition probability matrix, an Katz similarity matrix, among others. Approaches to factorize the representative matrix vary base on the matrix properties. If the obtaine matrix is positive semiefinite, e.g. the Laplacian matrix, one can use eigenvalue ecomposition. For unstructure matrices, one can use graient escent methos to obtain the embeing in linear time Locally Linear Embeing (LLE) LLE [26] assumes that every noe is a linear combination of its neighbors in the embeing space. If we assume that the ajacency matrix element W i j of graph G represents the weight of noe j in the representation of noe i, we efine Y i W i j Y j i V. j Hence, we can obtain the embeing Y N by minimizing φ(y) = Y i W i j Y j 2, i To remove egenerate solutions, the variance of the embeing is constraine as 1 N YT Y = I. To further remove translational invariance, the embeing is centere aroun zero: i Y i =. The above constraine optimization problem can be reuce to an eigenvalue problem, whose solution is to take the bottom + 1 eigenvectors of the sparse matrix (I W) T (I W) an iscaring the eigenvector corresponing to the smallest eigenvalue Laplacian Eigenmaps Laplacian Eigenmaps [2] aims to keep the embeing of two noes close when the weight W i j is high. Specifically, they minimize the following objective function φ(y) = 1 Y i Y j 2 W i j 2 i, j = tr(y T LY), where L is the Laplacian of graph G. The objective function is subjecte to the constraint Y T DY = I to eliminate trivial solution. The solution to this can be obtaine by taking the eigenvectors corresponing to the smallest eigenvalues of the normalize Laplacian, L norm = D 1/2 LD 1/ Cauchy Graph Embeing Laplacian Eigenmaps uses a quaratic penalty function on the istance between embeings. The objective function thus emphasizes preservation of issimilarity between noes more than their similarity. This may yiel embeings which o not preserve local topology, which can be efine as the equality between relative orer of ege weights (W i j ) an inverse orer of istances in the embee space ( Y i Y j 2 ). Cauchy Graph Embeing [32] tackles this problem by replacing the quaratic function Y i Y j 2 with Y i Y j 2 Y i Y j. Upon rearrangement, the objective function to be maximize becomes 2 +σ 2 W i j φ(y) = Y i Y j 2 + σ, 2 i, j with constraints Y T Y = I an i Y i = for each i. The new objective is an inverse function of istance an thus puts emphasis on similar noes rather than issimilar noes. The authors propose several variants incluing Gaussian, Exponential an Linear embeings with varying relative emphasis on the istance between noes. j

4 Category Year Publishe Metho Time Complexity Properties preserve 2 Science[26] LLE O( E 2 ) 21 NIPS[2] Laplacian Eigenmaps O( E 2 ) 1 st orer proximity Factorization 213 WWW[21] Graph Factorization O( E ) 21 CIKM[27] GraRep O( V 3 ) 216 KDD[24] HOPE O( E 2 ) 1 k th orer proximities 214 KDD[28] DeepWalk O( V ) Ranom Walk 216 KDD[29] noe2vec O( V ) 1 k th orer proximities, structural equivalence 216 KDD[23] SDNE O( V E ) 1 st an 2 n orer proximities Deep Learning 216 AAAI[3] DNGR O( V 2 ) 1 k th orer proximities 217 ICLR[31] GCN O( E 2 ) 1 k th orer proximities Miscellaneous 21 WWW[22] LINE O( E ) 1 st an 2 n orer proximities Table 1: List of graph embeing approaches G V E Y Graphical representation of the ata Set of vertices in the graph Set of eges in the graph Number of imensions Embeing of the graph, V Y i Embeing of noe v i, 1 (also i th row of Y) Y s Y t W D L Source embeing of a irecte graph, V Target embeing of a irecte graph, V Ajacency matrix of the graph, V V Diagonal matrix of the egree of each vertex, V V Graph Laplacian (L = D W), V V < Y i, Y j > Inner prouct of Y i an Y j i.e. Y i Y T j S Similarity matrix of the graph, V V Table 2: Summary of notation Structure Preserving Embeing (SPE) Structure Preserving Embeing ([33]) is another approach which extens Laplacian Eigenmaps. SPE aims to reconstruct the input graph exactly. The embeing is store as a positive semiefinite kernel matrix K an a connectivity algorithm G is efine which reconstructs the graph from K. The kernel K is chosen such that it maximizes tr(kw) which attempts to recover rank-1 spectral embeing. Choice of the connectivity algorithm G inuces constraints on this objective function. For e.g., if the connectivity scheme is to connect each noe to neighbors which lie within a ball of raius ɛ, the constraint (K ii + K j j 2K i j )(W i j 1/2) ɛ(w i j 1/2) prouces a kernel which can perfectly reconstruct the original graph. To hanle noise in the graph, a slack variable is ae. For ξ- connectivity, the optimization thus becomes max tr(ka) Cξ s.t. (K ii + K j j 2K i j )(W i j 1/2) ɛ(w i j 1/2) ξ, where ξ is the slack variable an C controls slackness Graph Factorization (GF) To the best of our knowlege, Graph Factorization [21] was the first metho to obtain a graph embeing in O( E ) time. To obtain the embeing, GF factorizes the ajacency matrix of the graph, minimizing the following loss function φ(y, λ) = 1 2 (W i j < Y i, Y j >) 2 + λ Y i 2, 2 (i, j) E where λ is a regularization coefficient. Note that the summation is over the observe eges as oppose to all possible eges. This is an approximation in the interest of scalability, an as such it may introuce noise in the solution. Note that as the ajacency matrix is often not positive semiefinite, the minimum of the loss function is greater than even if the imensionality of embeing is V GraRep GraRep [27] efines the noe transition probability as T = D 1 W an preserves k-orer proximity by minimizing X k Ys k Yt kt 2 F where Xk is erive from T k (refer to [27] for a etaile erivation). It then concatenates Ys k for all k to form Y s. Note that this is similar to HOPE [24] which minimizes S Y s Yt T 2 F where S is an appropriate similarity matrix. The rawback of GraRep is scalability, since T k can have O( V 2 ) non-zero entries HOPE HOPE [24] preserves higher orer proximity by minimizing S Y s Yt T 2 F, where S is the similarity matrix. The authors experimente with ifferent similarity measures, incluing Katz Inex, Roote Page Rank, Common Neighbors, an Aamic-Aar score. They represente each similarity measure as S = Mg 1 M l, where both M g an M l are sparse. This enables HOPE to use generalize Singular Value Decomposition (SVD) [34] to obtain the embeing efficiently. i 4

5 Aitional Variants For the purpose of imensionality reuction of high imensional ata, there are several other methos evelope capable of performing graph embeing. Yan et al. [3] survey a list of such methos incluing Principal Component Analysis (PCA) [36], Linear Discrimant Analysis (LDA) [37], ISOMAP [38], Multiimesional Scaling (MDS) [39], Locality Preserving Properties (LPP) [4] an Kernel Eigenmaps [41]. Matrinex et al. [42]) propose a general framework, non-negative graph embeing, which yiels non-negative embeings for these algorithms. A number of recent techniques have focuse on jointly learning network structure an aitional noe attribute information available for the network., Augmente Relation Embeing (ARE) [43] augments network with content base features for images an moifies the graph-laplacian to capture such information. Text-associate DeepWalk (TADW) [44] performs matrix factorization on noe similarity matrix isentangling the representation using text feature matrix. Heterogeneous Network Embeing (HNE) [4] learns representation for each moality of the network an then unifies them into a common space using linear transformations. Other works ([46, 47, 48]) perform similar transformations between various noe attributes an learn joint embeing Ranom Walk base Methos Ranom walks have been use to approximate many properties in the graph incluing noe centrality[49] an similarity[]. They are especially useful when one can either only partially observe the graph, or the graph is too large to measure in its entirety. Embeing techniques using ranom walks on graphs to obtain noe representations have been propose: DeepWalk an noe2vec are two examples DeepWalk DeepWalk [28] preserves higher-orer proximity between noes by maximizing the probability of observing the last k noes an the next k noes in the ranom walk centere at v i, i.e. maximizing log Pr(v i k,..., v i 1, v i+1,..., v i+k Y i ), where 2k + 1 is the length of the ranom walk. The moel generates multiple ranom walks each of length 2k + 1 an performs the optimization over sum of log-likelihoos for each ranom walk. A ot-prouct base ecoer is use to reconstruct the eges from the noe embeings noe2vec Similar to DeepWalk [28], noe2vec [29] preserves higherorer proximity between noes by maximizing the probability of occurrence of subsequent noes in fixe length ranom walks. The crucial ifference from DeepWalk is that noe2vec employs biase-ranom walks that provie a trae-off between breath-first (BFS) an epth-first (DFS) graph searches, an hence prouces higher-quality an more informative embeings than DeepWalk. Choosing the right balance enables noe2vec to preserve community structure as well as structural equivalence between noes Hierarchical Representation Learning for Networks (HARP) DeepWalk an noe2vec initialize the noe embeings ranomly for training the moels. As their objective function is non-convex, such initializations can be stuck in local optima. HARP [1] introuces a strategy to improve the solution an avoi local optima by better weight initialization. To this purpose, HARP creates hierarchy of noes by aggregating noes in the previous layer of hierarchy using graph coarsening. It then generates embeing of the coarsest graph an initializes the noe embeings of the refine graph (one up in the hierarchy) with the learne embeing. It propagates such embeings through the hierarchy to obtain the embeings of the original graph. Thus HARP can be use in conjunction with ranom walk base methos like DeepWalk an noe2vec to obtain better solutions to the optimization function Walklets DeepWalk an noe2vec implicitly preserve higher orer proximity between noes by generating multiple ranom walks which connect noes at various istances ue to its stochastic nature. On the other han, factorization base approches like GF an HOPE explicitly preserve istances between noes by moeling it in their objective function. Walklets [2] combine this iea of explicit moeling with ranom walks. The moel moifies the ranom walk strategy use in DeepWalk by sking over some noes in the graph. This is performe for multiple skip lengths, analogous to factorizing A k in GraRep, an the resulting set of ranom walks are use for training the moel similar to DeepWalk Aitional Variants There have been several variations of the above methos propose recently. Similar to augmenting graph structure with noe attributes for factorization base methos, GenVector [3], Discriminative Deep Ranom Walk (DDRW) [4], Tri-party Deep Network Representation (TriDNR) [] an [6] exten ranom walks to jointly learn network structure an noe attributes. 3.. Deep Learning base Methos The growing research on eep learning has le to a eluge of eep neural networks base methos applie to graphs[23, 3, 7]. Deep autoencoers have been use for imensionality reuction[8] ue to their ability to moel non-linear structure in the ata. Recently, SDNE [23], DNGR [3] utilize this ability of eep autoencoer to generate an embeing moel that can capture non-linearity in graphs Structural Deep Network Embeing (SDNE) Wang et al. [23] propose to use eep autoencoers to preserve the first an secon orer network proximities. They achieve this by jointly optimizing the two proximities. The approach uses highly non-linear functions to obtain the embeing. The moel consists of two parts: unsupervise an supervise. The former consists of an autoencoer aiming at fining

6 an embeing for a noe which can reconstruct its neighborhoo. The latter is base on Laplacian Eigenmaps[2] which apply a penalty when similar vertices are mappe far from each other in the embeing space Deep Neural Networks for Learning Graph Representations (DNGR) DNGR combines ranom surfing with eep autoencoer. The moel consists of 3 components: ranom surfing, positive pointwise mutual information (PPMI) calculation an stacke enoising autoencoers. Ranom surfing moel is use on the input graph to generate a probabilistic co-occurence matrix, analogous to similarity matrix in HOPE. The matrix is transforme to a PPMI matrix an input into a stacke enoising autoencoer to obtain the embeing. Inputting PPMI matrix ensures that the autoencoer moel can capture higher orer proximity. Furthermore, using stacke enoising autoencoers ais robustness of the moel in presence of noise in the graph as well as in capturing unerlying structure require for tasks such as link preiction an noe classification Graph Convolutional Networks (GCN) Deep neural network base methos iscusse above, namely SDNE an DNGR, take as input the global neighborhoo of each noe (a row of PPMI for DNGR an ajacency matrix for SDNE). This can be computationally expensive an inoptimal for large sparse graphs. Graph Convolutional Networks (GCNs) [31] tackle this problem by efining a convolution operator on graph. The moel iteratively aggregates the embeings of neighbors for a noe an uses a function of the obtaine embeing an its embeing at previous iteration to obtain the new embeing. Aggregating embeing of only local neighborhoo makes it scalable an multiple iterations allows the learne embeing of a noe to characterize global neighborhoo. Several recent papers ([9, 6, 61, 62, 63, 64]) have propose methos using convolution on graphs to obtain semisupervise embeing, which can be use to obtain unsupervise embeing by efining unique labels for each noe. The approaches vary in the construction of convolutional filters which can broaly be categorize into spatial an spectral filters. Spatial filters operate irectly on the original graph an ajacency matrix whereas spectral filters operate on the spectrum of graph- Laplacian Variational Graph Auto-Encoers (VGAE) Kipf et al. [6] evaluate the performance of variational autoencoers [66] on the task of graph embeing. The moel uses a graph convolutional network (GCN) encoer an an inner prouct ecoer. The input is ajacency matrix an they rely on GCN to learn the higher orer epenencies between noes. They empirically show that using variational autoencoers can improve performance compare to non-probabilistic autoencoers Other Methos LINE LINE [22] explicitly efines two functions, one each for first- an secon-orer proximities, an minimizes the combination of the two. The function for first-orer proximity is similar to that of Graph Factorization (GF) [21] in that they both aim to keep the ajacency matrix an ot prouct of embeings close. The ifference is that GF oes this by irectly minimizing the ifference of the two. Instea, LINE efines two joint probability istributions for each pair of vertices, one using ajancency matrix an the other using the embeing. Then, LINE minimizes the Kullback-Leibler (KL) ivergence of these two istributions. The two istributions an the objective function are as follows 1 p 1 (v i, v j ) = 1 + exp( < Y i, Y j >) W i j ˆp 1 (v i, v j ) = (i, j) E W i j O 1 = KL( ˆp 1, p 1 ) O 1 = W i j log p 1 (v i, v j ). (i, j) E The authors similarly efine probability istributions an objective function for the secon-orer proximity Discussion We can interpret embeings as representations which escribe graph ata. Thus, embeings can yiel insights into the properties of a network. We illustrate this in Figure 1. Consier a complete bipartite graph G. An embeing algorithm which attempts to keep two connecte noes close (i.e., preserve the community structure), woul fail to capture the structure of the graph as shown in 1(b). However, an algorithm which embes structurally-equivalent noes together learns an interpretable embeing as shown in 1(c). Similarly, in 1() we consier a graph with two star components connecte through a hub. Noes 1 an 3 are structurally equivalent (they link to the same noes) an are clustere together in 1(f), whereas in 1(e) they are far apart. The classes of algorithms above can be escribe in terms of their ability to explain the properties of graphs. Factorization-base methos are not capable of learning an arbitrary function, e.g., to explain network connectivity. Thus, unless explicitly inclue in their objective function, they cannot learn structural equivalence. In ranom walk base methos, the mixture of equivalences can be controlle to a certain extent by varying the ranom walk parameters. Deep learning methos can moel a wie range of functions following the universal approximation theorem [67]: given enough parameters, they can learn the mix of community an structural equivalence, to embe the noes such that the reconstruction error is minimize. We can interpret the weights of the autoencoer as a representation of the structure of the graph. For example, 1(c) plots the embeing learne by SDNE for the complete bipartite graph G 1. The autoencoer store the bipartite structure in

7 (a) Graph G 1 (b) CPE for G 1 (c) SPE for G 1 () Graph G 2 (e) CPE for G 2 (f) SPE for G 2 Figure 1: Examples illustrating the effect of type of similarity preserve. Here, CPE an SPE stan for Community Preserving Embeing an Structural-equivalence Preserving Embeing, respectively. weights an achieve perfect reconstruction. Given the variety of properties of real-worl graphs, using general non-linear moels that span a large class of functions is a promising irection that warrants further exploration. the original graph from the embeing an evaluating the reconstruction error. They show that a low imensional representation for each noe (in the orer of s) suffices to reconstruct the graph with high precision. 4. Applications As graph representations, embeings can be use in a variety of tasks. These applications can be broaly classifie as: network compression ( 4.1), visualization ( 4.2), clustering ( 4.3), link preiction ( 4.4), an noe classification ( 4.) Network Compression Feer et al.[68] introuce the concept of network compression (a.k.a. graph simplification). For a graph G, they efine a compression G which has smaller number of eges. The goal was to store the network more efficiently an run graph analysis algorithms faster. They obtaine the compression graph by partitioning the original graph into bipartite cliques an replacing them by trees, thus reucing the number of eges. Over the years, many researchers have use aggregation base methos [69, 7, 71] to compress graphs. The main iea in this line of work is to exploit the link structure of the graph to group noes an eges. Navlakha et al.[72] use Minimum Description Length (MDL) [73] from information theory to summarize a graph into a graph summary an ege correction. Similar to these representations, graph embeing can also be interprete as a summarization of graph. Wang et al.[23] an Ou et al.[24] teste this hypothesis explicitly by reconstructing 4.2. Visualization Application of visualizing graphs can be ate back to 1736 when Euler use it to solve Konigsberger Bruckenproblem [74]. In the recent years, graph visualization has foun applications in software engineering [7], electrical circuits [76], biology [1] an sociology [2]. Battista et al.[76] an Eaes et al.[77] survey a range of methos use to raw graphs an efine aesthetic criteria for this purpose. Herman et al.[78] generalize this an view it from an information visualization perspective. They stuy an compare various traitional layouts use to raw graphs incluing tree-, 3D- an hyperbolic-base layouts. As embeing represents a graph in a vector space, imensionality reuction techniques like Principal Component Analysis (PCA) [79] an t-istribute stochastic neighbor embeing (t-sne) [8] can be applie on it to visualize the graph. The authors of DeepWalk [28] illustrate the gooness of their embeing approach by visualizing the Zachary s Karate Club network. The authors of LINE [22] visualize the DBLP coauthorship network, an showe that LINE is able to cluster together authors in the same fiel. The authors of SDNE [23] applie it on 2-Newsgroup ocument similarity network to obtain clusters of ocuments base on topics. 7

8 4.3. Clustering Graph clustering (a.k.a., network partitioning) can be of two types: (a) structure base, an (b) attribute base clustering. The former can be further ivie into two categories, namely community base, an structurally equivalent clustering. Structure-base methos [7, 2, 8], aim to fin ense subgraphs with high number of intra-cluster eges, an low number of inter-cluster eges. Structural equivalence clustering [81], on the contrary, is esigne to ientify noes with similar roles (like briges an outliers). Attribute base methos [19] utilize noe labels, in aition to observe links, to cluster noes. White et al.[82] use k-means on the embeing to cluster the noes an visualize the clusters obtaine on Wornet an NCAA ata sets verifying that the clusters obtaine have intuitive interpretation. Recent methos on embeing haven t explicitly evaluate their moels on this task an thus it is a promising fiel of research in the graph embeing community. as noe classification. Bhagat et al.[6] survey the methos use in the literature for this task. They classify the approaches into two categories, i.e., feature extraction base an ranom walk base. Feature-base moels [11, 12, 87] generate features for noes base on their neighborhoo an local network statistics an then apply a classifier like Logistic Regression [88] an Naive Bayes [89] to preict the labels. Ranom walk base moels [9, ] propagate the labels with ranom walks. Embeings can be interprete as automatically extracte noe features base on network structure an thus falls into the first category. Recent work[28, 22, 24, 23, 29] has evaluate the preictive power of embeing on various information networks incluing language, social, biology an collaboration graphs. They show that embeings can preict missing labels with high precision.. Experimental Setup 4.4. Link Preiction Networks are constructe from the observe interactions between entities, which may be incomplete or inaccurate. The challenge often lies in ientifying spurious interactions an preicting missing information. Link preiction refers to the task of preicting either missing interactions or links that may appear in the future in an evolving network. Link preiction is pervasive in biological network analysis, where verifying the existence of links between noes requires costly experimental tests. Limiting the experiments to links orere by presence likelihoo has been shown to be very cost effective. In social networks, link preiction is use to preict probable frienships, which can be use for recommenation an lea to a more satisfactory user experience. Liben-Nowell et al.[], Lu et al.[83] an Hasan et al.[84] survey the recent progress in this fiel an categorize the algorithms into (a) similarity base (local an global) [13, 14, 8], (b) maximum likelihoo base [1, 16] an (c) probabilistic methos [17, 18, 86]. Embeings capture inherent ynamics of the network either explicitly or implicitly thus enabling application to link preiction. Wang et al.[23] an Ou et al.[24] preict links from the learne noe representations on publicly available collaboration an social networks. In aition, Grover et al.[29] apply it to biology networks. They show that on these ata sets links preicte using embeings are more accurate than traitional similarity base link preiction methos escribe above. 4.. Noe Classification Often in networks, a fraction of noes are labele. In social networks, labels may inicate interests, beliefs, or emographics. In language networks, a ocument may be labele with topics or keywors, whereas the labels of entities in biology networks may be base on functionality. Due to various factors, labels may be unknown for large fractions of noes. For example, in social networks many users o not provie their emographic information ue to privacy concerns. Missing labels can be inferre using the labele noes an the links in the network. The task of preicting these missing labels is also known 8 Our experiments evaluate the feature representations obtaine using the methos reviewe before on the previous four application omains. Next, we specify the atasets an evaluation metrics we use. The experiments were performe on a Ubuntu LTS system with 32 cores, 128 GB RAM an a clock spee of 2.6 GHz. The GPU use for eep network base moels was Nviia Tesla K4C..1. Datasets We evaluate the embeing approaches on a synthetic an 6 real atasets. The atasets are summarize in Table 3. SYN-SBM: We generate synthetic graph using Stochastic Block Moel [9] with 24 noes an 3 communities. We set the in-block an cross-block probabilities as.1 an 1 respectively. As we know the community structure in this graph, we use it to visualize the embeings learnt by various approaches. KARATE [91]: Zachary s karate network is a well-known social network of a university karate club. It has been wiely stuie in social network analysis. The network has 34 noes, 78 eges an 2 communities. BLOGCATALOG [92]: This is a network of social relationships of the bloggers liste on the BlogCatalog website. The labels represent blogger interests inferre through the metaata provie by the bloggers. The network has,312 noes, 333,983 eges an 39 ifferent labels. YOUTUBE [93]: This is a social network of Youtube users. This is a large network containing 1,17,827 noes an 4,94,382 eges. The labels represent groups of users who enjoy common vieo genres. HEP-TH [94]: The original ataset contains abstracts of papers in High Energy Physics Theory for the perio from January 1993 to April 23. We create a collaboration network for the papers publishe in this perio. The network has 7,98 noes an 21,36 eges. ASTRO-PH [9]: This is a collaboration network of authors of papers submitte to e-print arxiv uring the perio from January 1993 to April 23. The network has 18,772 noes an 396,16 eges.

9 Synthetic Social Network Collaboration Network Biology Network Name SYN-SBM KARATE BLOGCATALOG YOUTUBE HEP-TH ASTRO-PH PPI V 24 34,312 1,17,827 7,98 18,772 3,89 E 29, ,983 4,94,382 21,36 396,16 38,739 Avg. egree No. of labels Table 3: Dataset Statistics PROTEIN-PROTEIN INTERACTIONS (PPI) [96]: This is a network of biological interactions between proteins in humans. This network has 3,89 noes an 38,739 eges..2. Evaluation Metrics To evaluate the performance of embeing methos on graph reconstruction an link preiction, we use Precision at k (Pr@k) an MeanAveragePrecision(MAP) as our metrics. For noe classification, we use micro-f1 an macro-f1. These metrics are efine as follows: Pr@k is the fraction of correct preictions in top k preictions. It is efine as Pr@k = E pre(1:k) E obs k, where E pre (1 : k) are the top k preictions an E obs are the observe eges. For the task of graph reconstruction, E obs = E an for link preiction, E obs is the set of hien eges. MAP estimates precision for every noe an computes the average over all noes, as follows: i AP(i) MAP =, V k Pr@k(i) I{E where AP(i) = prei (k) E obsi } {k:e prei (k) E obsi }, Pr@k(i) = E pre i (1:k) E obsi k, an E prei an E obsi are the preicte an observe eges for noe i respectively. macro-f1, in a multi-label classification task, is efine as the average F1 of all the labels, i.e., l L F1(l) macro F1 =, L where F1(l) is the F1-score for label l. micro-f1 calculates F1 globally by counting the total true positives, false negatives an false positives, giving equal weight to each instance. It is efine as follows: micro F1 = 2 P R P + R, l L T P(l) where P = l L(T P(l)+FP(l)), an R = l L T P(l) l L(T P(l)+FN(l)), are precision (P) an recall (R) respectively, an T P(l), FP(l) an FN(l) enote the number of true positives, false positives an false negatives respectively among the instances which are associate with the label l either in the groun truth or the preictions. 6. Experiments an Analysis In this section, we evaluate an compare embeing methos on the for tasks presente above. For each task, we show 9 the effect of number of embeing imensions on the performance an compare hyper parameter sensitivity of the methos. Furthermore, we correlate the performance of embeing techniques on various tasks varying hyper parameters to test the notion of an all-goo embeing which can perform well on all tasks Graph Reconstruction Embeings as a low-imensional representation of the graph are expecte to accurately reconstruct the graph. Note that reconstruction iffers for ifferent embeing techniques (refer to Section 3). For each metho, we reconstruct the proximity of noes an rank pair of noes accoring to their proximity. Then we calculate the ratio of real links in top k preictions as the reconstruction precision. As the number of possible noe pairs (N(N 1)) can be very large for networks with a large number of noes, we ranomly sample 24 noes for evaluation. We obtain such samples for each ataset an calculate the mean an stanar eviation of precision an MAP values for subgraph reconstruction. To obtain optimal hyperparameters for each embeing metho, we compare the mean MAP values for each hyperparameter. We then re-run the experiments with the optimal hyperparameter on ranom samples of 24 noes an report the results. Figure 2 illustrates the reconstruction precision obtaine by 128-imensional embeings. We observe that although performance of methos is ataset epenent, embeing approaches which preserve higher orer proximities in general outperform others. Exceptional performance of Laplacian Eigenmaps on SBM can be attribute to the lack of higher orer structure in the ata set. We also observe that SDNE consistently performs well on all ata sets. This can be attribute to its capability of learning complex structure from the network. Embeings learne by noe2vec have low reconstruction precision. This may be ue to the highly non-linear imensionality reuction yieling a non-linear manifol. However, HOPE, which learns linear embeings but preserves higher orer proximity reconstructs the graph well without any aitional parameters. Effect of imension. Figure 3 illustrates the effect of imension on the reconstruction error. With a couple of exceptions, as the number of imensions increase, the MAP value increases. This is intuitive as higher number of imensions are capable of storing more information. We also observe that SDNE is able to embe the graphs in 16-imensional vector space with high precision although ecoer parameters are require to obtain such precision.

10 SBM PPI AstroPh BlogCatalog Hep-th Youtube precision@k k k k noe2vec GF SDNE HOPE LE Figure 2: Precision@k of graph reconstruction for ifferent ata sets (imension of embeing is 128) Visualization Since embeing is a low-imensional vector representation of noes in the graph, it allows us to visualize the noes to unerstan the network topology. As ifferent embeing methos preserve ifferent structures in the network, their ability an interpretation of noe visualization iffer. For instance, embeings learne by noe2vec with parameters set to prefer BFS ranom walk woul cluster structurally equivalent noes together. On the other han, methos which irectly preserve k-hop istances between noes (GF, LE an LLE with k = 1 an HOPE an SDNE with k > 1) cluster neighboring noes together. We compare the ability of ifferent methos to visualize noes on SBM an Karate graph. For SBM, following [23], we learn a 128-imensional embeing for each metho an input it to t-sne [8] to reuce the imensionality to 2 an visualize noes in a 2-imensional space. Visualization of SBM is show in Figure 4. As we know the unerlying community structure, we use the community label to color the noes. We observe that embeings generate by HOPE an SDNE which preserve higher orer proximities well separate the communities although as the ata is well structure LE, GF an LLE are able to capture community structure to some extent. We visualize Karate graph (see Figure ) to illustrate the properties preserve by embeing methos. LLE an LE ((a) an (f)) attempt to preserve the community structure of the graph an cluster noes with high intra-cluster eges together. GF ((b)) embes communities very closely an keeps leaf noes far away from other noes. In (), we observe that HOPE embes noes 16 an 21, whose Katz similarity in the original graph is very low (6), farthest apart (consiering ot prouct similarity). noe2vec an SDNE ((c) an (e)) preserve a mix of community structure an structural property of the noes. Noes 32 an 33, which are both high egree hubs an central in their communities, are embee together an away from low egree noes. Also, they are closer to noes which belong to their communities. SDNE embes noe, which acts a brige between communities, far away from other noes. Note that, unlike for other methos, it oes not imply that noe is isconnecte from the rest of the noes. The implication here is that SDNE ientifies noe as a separate type of noe an encoes its connection to other noes in encoer an ecoer. The ability of eep autoencoers to ientify important noes in the network has not been stuie but given this observation we believe this irection can be promising Link Preiction Another important application of graph embeing is preicting unobserve links in the graph. A goo network representation shoul be able to capture the inherent structure of graph well enough to preict the likely but unobserve links. To test the performance of ifferent embeing methos on this

11 SBM PPI AstroPh.7.. MAP.1.1 MAP BlogCatalog Hep-th Youtube noe2vec GF SDNE HOPE LE Figure 3: MAP of graph reconstruction for ifferent ata sets with varying imensions (a) LLE 1 1 (b) GF (c) noe2vec () HOPE (e) SDNE (f) LE Figure 4: Visualization of SBM using t-sne (original imension of embeing is 128). Each point correspons to a noe in the graph. Color of a noe enotes its community. 11

12 (a) LLE (b) GF (c) noe2vec () HOPE (e) SDNE (f) LE Figure : Visualization of Karate club graph. Each point correspons to a noe in the graph. PPI AstroPh BlogCatalog Hep-th precision@k k noe2vec k GF SDNE k HOPE LE k Figure 6: Precision@k of link preiction for ifferent ata sets (imension of embeing is 128). PPI AstroPh BlogCatalog Hep-th MAP noe2vec GF SDNE HOPE LE Figure 7: MAP of link preiction for ifferent ata sets with varying imensions. 12

13 task, for each ata set we ranomly hie 2% of the network eges. We learn the embeing using the rest of the 8% eges an preict the most likely eges which are not observe in the training ata from the learne embeing. As with graph reconstruction, we generate ranom subgraphs with 24 noes an test the preicte links against the hel-out links in the subgraphs. We perform this experiment for each hyperparameter an re-run it for optimal hyperparameters on a new ranom 8-2 link split. Figure 6 an 7 show the precision@k results for link preiction with 128-imensional embeings an MAP for each imension respectively. Here we can see that the performance of methos is highly ata set epenent. noe2vec achieves goo performance on BlogCatalog but performs poorly on other ata sets. HOPE achieves goo performance on all ata sets which implies that preserving higher orer proximities is conucive to preicting unobserve links. Similarly, SDNE outperforms other methos with the exception on PPI for which the performance egraes rastically as embeing imension increases above 8. Effect of imension. Figure 7 illustrates the effect of embeing imension on link preiction. We make two observations. Firstly, in PPI an BlogCatalog, unlike graph reconstruction performance oes not improve as the number of imensions increase. This may be because with more parameters the moels overfit on the observe links an are unable to preict unobserve links. Seconly, even on the same ata set, relative performance of methos epens on the embeing imension. In PPI, HOPE outperforms other methos for higher imensions,whereas embeing generate by SDNE achieves higher link preiction MAP for low imensions Noe Classification Preicting noe labels using network topology is wiely popular in network analysis an has variety of applications, incluing ocument classification an interest preiction. A goo network embeing shoul capture the network structure an hence be useful for noe classification. We compare the effectiveness of embeing methos on this task by using the generate embeing as noe features to classify the noes. The noe features are input to a one-vs-rest logistic regression using the LIBLINEAR library. For each ata set, we ranomly sample % to 9% of noes as training ata an evaluate the performance on the remaining noes. We perform this split times an report the mean with confience interval. For ata sets with multiple labels per noe, we assume that we know how many labels to preict. Figure 8 shows the results of our experiments. We can see that noe2vec outperforms other methos on the task of noe classification. As mentione earlier ( 3), noe2vec preserves homophily as well as structural equivalence between noes. Results suggest this can be useful in noe classification: e.g., in BlogCatalog users may have similar interests, yet connect to others base on social ties rather than interests overlap. Similarly, proteins in PPI may be relate in functionality an interact with similar proteins but may not assist each other. However, in SBM, other methos outperform noe2vec as labels reflect communities yet there is no structural equivalence between noes. Effect of imension. Figure 9 illustrates the effect of embeing imensions on noe classification. As with link preiction, we observe that performance often saturates or eteriorates after certain number of imensions. This may suggest overfitting on the training ata. As SBM exhibits very structure communities, an 8-imensional embeing suffices to preict the communities. noe2vec achieves best performance on PPI an BlogCat with 128 imensions. 6.. Hyperparameter Sensitivity In this section we plan to aress the following questions: How robust are the embeing methos with respect to hyperparameters? Do optimal hyperparameters epen on the ownstream tasks the embeings are use for? What insights oes performance variance with hyperparameter provie about a ata set? We answer these questions by analyzing performance of the embeing methos with various hyperparameters. We present results on SBM, PPI an Hep-th. Of the methos we evaluate in this survey, Laplacian Eigenmaps has no hyperparameters, thus is not inclue in the following analysis. Graph Factorization (GF). The objective function of GF contains a weight regularization term with a coefficient. Intuitively, this coefficient controls the generalizability of the embeing. A low regularization coefficient facilitates better reconstruction but may overfit to the observe graph leaing to poor preiction performance. On the other sie, a high regularization may unerrepresent the ata an hence perform poorly on all tasks. We observe this effect in Figure. We see that performance on preiction tasks, namely link preiction (Fig. b) an noe classification (Fig. c), improves as we increase the regularization coefficient, reaches a peak an then starts eteriorating. However, graph reconstruction performance (Fig. a) may eteriorate with increasing regularization. We also note that the performance change is consierable an thus the coefficient shoul be carefully tune to the given ata set. HOPE. As HOPE factorizes a similarity matrix between noes to obtain the embeing, the hyperparameters epen on the metho use to obtain the similarity matrix. Since in our experiments we use Katz inex for this purpose, we evaluate the effect of the attenuation factor (β), which can be interprete as the higher orer proximity coefficient on performance. Graph structure affects the optimal value of this parameter. For well-structure graphs with tightly knit communities, high values of beta woul erroneously assign issimilar noes closer in the embeing space. On the contrary, for graphs with weak community structure it is important to capture higher orer istances an thus high values of beta may yiel better embeings. We valiate our hypothesis in Figure 11. As our synthetic ata SBM consists of tight communities, increasing β oes not improve the performance on any task. However, gain