Deep Learning on Graphs

Transcription

1 Deep Learning on Graphs with Graph Convolutional Networks Hidden layer Hidden layer Input Output ReLU ReLU, 22 March 2017 joint work with Max Welling (University of Amsterdam) BDL NIPS 2016 Conference ICLR 2017 (+ new paper on arxiv)

2 The success story of deep learning Speech data Natural language processing (NLP) 2

3 The success story of deep learning Speech data Natural language processing (NLP) Deep neural nets that exploit: - translation invariance (weight sharing) - hierarchical compositionality 2

4 Recap: Deep learning on Euclidean data Euclidean data: grids, sequences 2D grid 3

5 Recap: Deep learning on Euclidean data Euclidean data: grids, sequences 2D grid 1D grid 3

6 Recap: Deep learning on Euclidean data We know how to deal with this: Convolutional neural networks (CNNs) (Animation by Vincent Dumoulin) (Source: Wikipedia) 4

7 Recap: Deep learning on Euclidean data We know how to deal with this: Convolutional neural networks (CNNs) (Animation by Vincent Dumoulin) (Source: Wikipedia) 4

8 Recap: Deep learning on Euclidean data We know how to deal with this: Convolutional neural networks (CNNs) (Animation by Vincent Dumoulin) (Source: Wikipedia) or recurrent neural networks (RNNs) (Source: Christopher Olah s blog) 4

9 Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) 5

10 Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) 5

11 Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) 5

12 Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) Update for a single pixel: Transform neighbors individually Add everything up 5

13 Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) Update for a single pixel: Transform neighbors individually Add everything up Full update: 5

14 Graph-structured data What if our data looks like this? 6

15 Graph-structured data What if our data looks like this? or this: 6

16 Graph-structured data What if our data looks like this? or this: Real-world examples: Social networks World-wide-web Protein-interaction networks Telecommunication networks Knowledge graphs 6

17 Graphs: Definitions Graph: : Set of nodes, : Set of edges We can define: (adjacency matrix): (can also be weighted) 7

18 Graphs: Definitions Graph: : Set of nodes, : Set of edges We can define: (adjacency matrix): (can also be weighted) Model wish list: Set of trainable parameters Trainable in time Applicable even if the input graph changes 7

19 A naive approach Take adjacency matrix and feature matrix Concatenate them Feed them into deep (fully connected) neural net Done?? 8

20 A naive approach Take adjacency matrix and feature matrix Concatenate them Feed them into deep (fully connected) neural net Done?? Problems: Huge number of parameters Needs to be re-trained if number of nodes changes Does not generalize across graphs 8

21 A naive approach Take adjacency matrix and feature matrix Concatenate them Feed them into deep (fully connected) neural net Done?? Problems: Huge number of parameters Needs to be re-trained if number of nodes changes Does not generalize across graphs 8

22 A naive approach Take adjacency matrix and feature matrix Concatenate them Feed them into deep (fully connected) neural net Done? We need weight sharing! => CNNs on graphs or Graph Convolutional Networks (GCNs)? Problems: Huge number of parameters Needs to be re-trained if number of nodes changes Does not generalize across graphs 8

23 CNNs on graphs with spatial filters Consider this undirected graph: (related idea was first proposed in Scarselli et al. 2009) 9

24 CNNs on graphs with spatial filters (related idea was first proposed in Scarselli et al. 2009) Consider this undirected graph: Calculate update for node in red: 9

25 CNNs on graphs with spatial filters (related idea was first proposed in Scarselli et al. 2009) Consider this undirected graph: Calculate update for node in red: 9

26 CNNs on graphs with spatial filters (related idea was first proposed in Scarselli et al. 2009) Consider this undirected graph: Calculate update for node in red: Update rule: : neighbor indices : norm. constant (per edge) 9

27 CNNs on graphs with spatial filters (related idea was first proposed in Scarselli et al. 2009) Consider this undirected graph: Calculate update for node in red: Update rule: : neighbor indices : norm. constant (per edge) How is this related to spectral CNNs on graphs? Localized 1st-order approximation of spectral filters [Kipf & Welling, ICLR 2017] 9

28 Vectorized form with 10

29 Vectorized form with Or treat self-connection in the same way: with 10

30 Vectorized form with Or treat self-connection in the same way: with is typically sparse We can use sparse matrix multiplications! Efficient implementation in Theano or TensorFlow 10

31 Model architecture (with spatial filters) Input: Feature matrix, preprocessed adjacency matrix Hidden layer Hidden layer Input Output ReLU ReLU (or more sophisticated filters / basis functions) [Kipf & Welling, ICLR 2017] 11

32 What does it do? An example. Forward pass through untrained 3-layer GCN model Parameters initialized randomly 2-dim output per node f( ) = [Karate Club Network] Produces (useful?) random embeddings! 12

33 Connection to Weisfeiler-Lehman algorithm A classical approach for node feature assignment 13

34 Connection to Weisfeiler-Lehman algorithm A classical approach for node feature assignment 13

35 Connection to Weisfeiler-Lehman algorithm A classical approach for node feature assignment Useful as graph isomorphism check for most graphs (exception: highly regular graphs) 13

36 Connection to Weisfeiler-Lehman algorithm A classical approach for node feature assignment Useful as graph isomorphism check for most graphs (exception: highly regular graphs) 13

37 Connection to Weisfeiler-Lehman algorithm A classical approach for node feature assignment GCN: Useful as graph isomorphism check for most graphs (exception: highly regular graphs) 13

38 Semi-supervised classification on graphs Setting: Some nodes are labeled (black circle) All other nodes are unlabeled Task: Predict node label of unlabeled nodes 14

39 Semi-supervised classification on graphs Setting: Some nodes are labeled (black circle) All other nodes are unlabeled Task: Predict node label of unlabeled nodes Standard approach: graph-based regularization ( smoothness constraints ) [Zhu et al., 2003] with assumes: connected nodes likely to share same label 14

40 Semi-supervised classification on graphs Embedding-based approaches Two-step pipeline: 1) Get embedding for every node 2) Train classifier on node embedding Examples: DeepWalk [Perozzi et al., 2014], node2vec [Grover & Leskovec, 2016] 15

41 Semi-supervised classification on graphs Embedding-based approaches Two-step pipeline: 1) Get embedding for every node 2) Train classifier on node embedding Examples: DeepWalk [Perozzi et al., 2014], node2vec [Grover & Leskovec, 2016] Problem: Embeddings are not optimized for classification! 15

42 Semi-supervised classification on graphs Embedding-based approaches Two-step pipeline: 1) Get embedding for every node 2) Train classifier on node embedding Examples: DeepWalk [Perozzi et al., 2014], node2vec [Grover & Leskovec, 2016] Problem: Embeddings are not optimized for classification! Idea: Train graph-based classifier end-to-end using GCN Evaluate loss on labeled nodes only: set of labeled node indices label matrix GCN output (after softmax) 15

43 Toy example (semi-supervised learning) Video also available here: 16

44 Toy example (semi-supervised learning) Video also available here: 16

45 Classification on citation networks Input: Citation networks (nodes are papers, edges are citation links, optionally bag-of-words features on nodes) Target: Paper category (e.g. stat.ml, cs.lg, ) (Figure from: Bronstein, Bruna, LeCun, Szlam, Vandergheynst, 2016) 17

46 Experiments and results Model: 2-layer GCN Dataset statistics (Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017) 18

47 Experiments and results Model: 2-layer GCN Dataset statistics Classification results (accuracy) no input features (Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017) 18

48 Experiments and results Model: 2-layer GCN Dataset statistics no input features Classification results (accuracy) Learned representations (t-sne embedding of hidden layer activations) (Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017) 18

49 Other recent work Global filters (hard to generalize) Spectral domain Local filters (limited field of view) Spatial domain SyncSpecCNN (Spectral Transformer Nets) [Yi et al., 2016] Mixture model CNN (learned basis functions) [Monti et al., 2016] Chebyshev CNN (approximate spectral filters in spatial domain) [Defferrard et al., 2016] 19

50 Conclusions and open questions Results from past 1-2 years have shown: Deep learning paradigm can be extended to graphs State-of-the-art results in a number of domains Use end-to-end training instead of multi-stage approaches (graph kernels, DeepWalk, node2vec etc.) Open problems: Learn basis functions from scratch - First step: [Monti et al., 2016] Robustness of filters to change of graph structure Learn representations of graphs - First step: [Duvenaud et al., 2015] Theoretical understanding of what is learned Common benchmark datasets 20

51 Further reading Blog post Graph Convolutional Networks: Code on Github: Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017: Kipf & Welling, Variational Graph Auto-Encoders, NIPS BDL Workshop, 2016: You can get in touch with me via: Web: Project funded by SAP 21

52 How deep is deep enough? Residual connection 22