Blind Identification of Graph Filters with Multiple Sparse Inputs

Transcription

1 Blind Identification of Graph Filters with Multiple Sparse Inputs Santiago Segarra, Antonio G. Marques, Gonzalo Mateos & Alejandro Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania ICASSP, March 24, 2016 Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 1 / 22

2 Network Science analytics Online social media Internet Clean energy and grid analy,cs Desiderata: Process, analyze and learn from network data [Kolaczyk 09] Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 2 / 22

3 Network Science analytics Online social media Internet Clean energy and grid analy,cs Desiderata: Process, analyze and learn from network data [Kolaczyk 09] Network as graph G = (V, E): encode pairwise relationships Interest here not in G itself, but in data associated with nodes in V The object of study is a graph signal Ex: Opinion profile, buffer congestion levels, neural activity, epidemic Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 2 / 22

4 Motivating examples Graph signals Graph SP: broaden classical SP to graph signals [Shuman etal 13] Our view: GSP well suited to study network processes As.: Signal properties related to topology of G (e.g., smoothness) Algorithms that fruitfully leverage this relational structure Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 3 / 22

5 Graph signals Consider a graph G(V, E). Graph signals are mappings x : V R Defined on the vertices of the graph (data tied to nodes) May be represented as a vector x R N x n denotes the signal value at the n-th vertex in V Implicit ordering of vertices x x x = x 2 = x Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 4 / 22

6 Graph-shift operator To understand and analyze x, useful to account for G s structure Graph G is endowed with a graph-shift operator S R N N S ij = 0 for i j and (i, j) E (captures local structure in G) S can take nonzero values in the edges of G or in its diagonal Ex: Adjacency A, degree D, and Laplacian L = D A matrices Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 5 / 22

7 Locality of the graph-shift operator S is a linear operator that can be computed locally at the nodes in V Consider the graph signal y = Sx and node i s neighborhood N i Node i can compute y i if it has access to x j at j N i y i = S ij x j, i V j N i Recall S ij 0 only if i = j or (j, i) E If y = S 2 x y i found using values x j within 2 hops Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 6 / 22

8 Graph Fourier transform (GFT) As.: S related to generation (description) of the signals of interest Spectrum of S = VΛV 1 will be especially useful to analyze x The Graph Fourier Transform (GFT) of x is defined as x = V 1 x While the inverse GFT (igft) of x is defined as x = V x Eigenvectors V = [v 1,..., v N ] are the frequency basis (atoms) Ex: For the directed cycle (temporal signal) GFT DFT Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 7 / 22

9 Linear (shift-invariant) graph filter A graph filter H : R N R N is a map between graph signals Focus on linear filters map represented by an N N matrix Polynomial in S of degree L, with coefficients h = [h 0,..., h L ] T Graph filter [Sandryhaila-Moura 13] H := h 0 S 0 + h 1 S h L S L = L l=0 h ls l Key properties: shift-invariance and distributed implementation H(Sx) = S(Hx), no other can be linear and shift-invariant Each application of S only local info only L-hop info for y = Hx Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 8 / 22

10 Frequency response of a graph filter Using S = VΛV 1, filter is H = ( L l=0 h L ) ls l = V l=0 h lλ l V 1 Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 9 / 22

11 Frequency response of a graph filter Using S = VΛV 1, filter is H = ( L l=0 h L ) ls l = V l=0 h lλ l V 1 Since Λ l are diagonal, use GFT-iGFT to write y = Hx as ỹ = diag( h) x Output at frequency k depends only on input at frequency k Frequency response of the filter H is h = Ψh, with Vandermonde Ψ Ψ := 1 λ 1... λ L λ N... λ L N Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 9 / 22

12 Frequency response of a graph filter Using S = VΛV 1, filter is H = ( L l=0 h L ) ls l = V l=0 h lλ l V 1 Since Λ l are diagonal, use GFT-iGFT to write y = Hx as ỹ = diag( h) x Output at frequency k depends only on input at frequency k Frequency response of the filter H is h = Ψh, with Vandermonde Ψ Ψ := 1 λ 1... λ L λ N... λ L N GFT for signals ( x = V 1 x) and filters ( h = Ψh) is different Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 9 / 22

13 Diffusion processes as graph filter outputs Q: Upon observing a graph signal y, how was this signal generated? Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 10 / 22

14 Diffusion processes as graph filter outputs Q: Upon observing a graph signal y, how was this signal generated? Postulate the following generative model An originally sparse signal x = x (0) Diffused via linear graph dynamics S x (l) = Sx (l 1) Observed y is a linear combination of the diffused signals x (l) y = L h l x (l) = l=0 L h l S l x = Hx l=0 Model: Observed network process as output of a graph filter View few elements in supp(x) =: {i : x i 0} as seeds Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 10 / 22

15 Motivation and problem statement Ex: Global opinion/belief profile formed by spreading a rumor What was the rumor? Who started it? How do people weigh in peers opinions to form their own? x y Graph Filter Unobserved Observed Problem: Blind identification of graph filters with sparse inputs Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 11 / 22

16 Motivation and problem statement Ex: Global opinion/belief profile formed by spreading a rumor What was the rumor? Who started it? How do people weigh in peers opinions to form their own? x y Graph Filter Unobserved Observed Problem: Blind identification of graph filters with sparse inputs Q: Given S, can we find x and the combination weights h from y = Hx? Extends classical blind deconvolution to graphs Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 11 / 22

17 Blind graph filter identification Leverage frequency response of graph filters (U := V 1 ) y = Hx y = Vdiag(Ψh)Ux y is a bilinear function of the unknowns h and x Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 12 / 22

18 Blind graph filter identification Leverage frequency response of graph filters (U := V 1 ) y = Hx y = Vdiag(Ψh)Ux y is a bilinear function of the unknowns h and x Problem is ill-posed (L + 1) + N unknowns and N observations As.: x is S-sparse i.e., x 0 := supp(x) S Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 12 / 22

19 Blind graph filter identification Leverage frequency response of graph filters (U := V 1 ) y = Hx y = Vdiag(Ψh)Ux y is a bilinear function of the unknowns h and x Problem is ill-posed (L + 1) + N unknowns and N observations As.: x is S-sparse i.e., x 0 := supp(x) S Blind graph filter identification Non-convex feasibility problem find {h, x}, s. to y = Vdiag ( Ψh ) Ux, x 0 S Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 12 / 22

20 Lifting the bilinear inverse problem Key observation: Use the Khatri-Rao product to write y as y = V(Ψ T U T ) T vec(xh T ) Reveals y is a linear combination of the entries of Z := xh T Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 13 / 22

21 Lifting the bilinear inverse problem Key observation: Use the Khatri-Rao product to write y as y = V(Ψ T U T ) T vec(xh T ) Reveals y is a linear combination of the entries of Z := xh T Z is of rank-1 and row-sparse Linear matrix inverse problem min Z rank(z), s. to y = V ( Ψ T U T ) T vec ( Z ), Z 2,0 S Pseudo-norm Z 2,0 counts the nonzero rows of Z Matrix lifting for blind deconvolution [Ahmed etal 14] Rank minimization s. to row-cardinality constraint is NP-hard Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 13 / 22

22 Lifting the bilinear inverse problem Key observation: Use the Khatri-Rao product to write y as y = V(Ψ T U T ) T vec(xh T ) Reveals y is a linear combination of the entries of Z := xh T Z is of rank-1 and row-sparse Linear matrix inverse problem min Z rank(z), s. to y = V ( Ψ T U T ) T vec ( Z ), Z 2,0 S Pseudo-norm Z 2,0 counts the nonzero rows of Z Matrix lifting for blind deconvolution [Ahmed etal 14] Rank minimization s. to row-cardinality constraint is NP-hard. Relax! Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 13 / 22

23 Algorithmic approach via convex relaxation Key property: l 1 -norm minimization promotes sparsity [Tibshirani 94] Nuclear norm Z := i σ i(z) a convex proxy of rank [Fazel 01] l2,1 norm Z 2,1 := i zt i 2 surrogate of Z 2,0 [Yuan-Lin 06] Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 14 / 22

24 Algorithmic approach via convex relaxation Key property: l 1 -norm minimization promotes sparsity [Tibshirani 94] Nuclear norm Z := i σ i(z) a convex proxy of rank [Fazel 01] l2,1 norm Z 2,1 := i zt i 2 surrogate of Z 2,0 [Yuan-Lin 06] Convex relaxation min Z Z + α Z 2,1, s. to y = V ( Ψ T U T ) T vec ( Z ) Scalable algorithm using method of multipliers Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 14 / 22

25 Algorithmic approach via convex relaxation Key property: l 1 -norm minimization promotes sparsity [Tibshirani 94] Nuclear norm Z := i σ i(z) a convex proxy of rank [Fazel 01] l2,1 norm Z 2,1 := i zt i 2 surrogate of Z 2,0 [Yuan-Lin 06] Convex relaxation min Z Z + α Z 2,1, s. to y = V ( Ψ T U T ) T vec ( Z ) Scalable algorithm using method of multipliers Refine estimates {h, x} via iteratively-reweighted optimization Weights α i (k) = ( z i (k) T 2 + δ) 1 per row i, per iteration k Exact recovery conditions Success of the convex relaxation Random model on the graph structure Recovery conditions Probabilistic guarantees that depend on the graph spectrum Blind deconvolution (in time) is a favorable graph setting Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 14 / 22

26 Numerical tests: Recovery rates Recovery rates over an (L, S) grid and 20 trials Successful recovery when x (h ) T xh T F < 10 3 ER (left), ER reweighted l 2,1 (center), brain reweighted l 2,1 (right) L S L S L S Exact recovery over non-trivial (L, S) region Reweighted optimization markedly improves performance Encouraging results even for real-world graphs Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 15 / 22

27 Error Numerical tests: Brain graph Human brain graph with N = 66 regions, L = 6 and S = LS AM RM RM (k.s.) Number of Observations Proposed method also outperforms alternating-minimization solver Unknown supp(x) Need twice as many observations Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 16 / 22

28 Multiple output signals Suppose we have access to P output signals {y p } P p=1 x 1 y 1 Graph Filter x P y P Graph Filter Unobserved Observed Goal: Identify common filter H fed by multiple unobserved inputs x p Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 17 / 22

29 Formulation As.: {x p } P p=1 are S-sparse with common support Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 18 / 22

30 Formulation As.: {x p } P p=1 are S-sparse with common support Concatenate outputs ȳ := [y1 T,..., yt P ]T and inputs x := [x T 1,..., xt P ]T Unknown rank-one matrices Z p := x p h T. Stack them Vertically in rank one Z v := [Z T 1,..., ZT P ]T = xh T R NP L Horizontally in row sparse Z h := [Z 1,..., Z P ] R N PL Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 18 / 22

31 Formulation As.: {x p } P p=1 are S-sparse with common support Concatenate outputs ȳ := [y T 1,..., yt P ]T and inputs x := [x T 1,..., xt P ]T Unknown rank-one matrices Z p := x p h T. Stack them Vertically in rank one Z v := [Z T 1,..., ZT P ]T = xh T R NP L Horizontally in row sparse Z h := [Z 1,..., Z P ] R N PL Convex formulation ( min Z v +τ Z h 2,1, s. to ȳ = I P (V ( Ψ T U T ) )) T vec ( Zh ) {Z p} P p=1 Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 18 / 22

32 Formulation As.: {x p } P p=1 are S-sparse with common support Concatenate outputs ȳ := [y T 1,..., yt P ]T and inputs x := [x T 1,..., xt P ]T Unknown rank-one matrices Z p := x p h T. Stack them Vertically in rank one Z v := [Z T 1,..., ZT P ]T = xh T R NP L Horizontally in row sparse Z h := [Z 1,..., Z P ] R N PL Convex formulation ( min Z v +τ Z h 2,1, s. to ȳ = I P (V ( Ψ T U T ) )) T vec ( Zh ) {Z p} P p=1 Relax (As.): Z h 2,1 Z v 2,1 = P p=1 Z p 2,1 Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 18 / 22

33 Numerical tests: Multiple signals, recovery rates Recovery rates over an (L, S) grid and 20 trials Successful recovery when ˆ xĥt xh T F < 10 3 ER (left), ER reweighted l 2,1 (center), brain reweighted l 2,1 (right) P=1 L 3 4 L 3 4 L S S S P=5 L 2 3 L 2 3 L S S S 0 Leveraging multiple output signals aids the blind identification task Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 19 / 22

34 Summary and extensions Extended blind deconvolution of space/time signals to graphs Key: model diffusion process as output of graph filter Rank and sparsity minimization subject to model constraints Lifting and convex relaxation yield efficient algorithms Exact recovery conditions Success of the convex relaxation Probabilistic guarantees that depend on the graph spectrum Consideration of multiple sparse inputs aids recovery Envisioned application domains (a) Opinion formation in social networks (b) Identify sources of epileptic seizure (c) Trace patient zero for an epidemic outbreak Unknown shift S Network topology inference Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 20 / 22

35 GlobalSIP 16 Symposium on Networks Symposium on Signal and Information Processing over Networks Topics of interest Graph-signal transforms and filters Signals in high-order graphs Non-linear graph SP Graph algorithms for network analytics Statistical graph SP Graph-based distributed SP algorithms Prediction and learning in graphs Graph-based image and video processing Network topology inference Communications, sensor and power networks Network tomography Neuroscience and other medical fields Control of network processes Web, economic and social networks Paper submission due: June 5, 2016 Organizers: Michael Rabbat (McGill Univ.) Antonio Marques (King Juan Carlos Univ.) Gonzalo Mateos (Univ. of Rochester) Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 21 / 22

36 Relevance of the graph-shift operator Q: Why is S called shift? Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 22 / 22

37 Relevance of the graph-shift operator Q: Why is S called shift? A: Resemblance to time shifts Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 22 / 22

38 Relevance of the graph-shift operator Q: Why is S called shift? A: Resemblance to time shifts S will be building block for GSP algorithms Same is true in the time domain (filters and delay) Santiago Segarra Blind Identification of Graph Filters with Multiple Sparse Inputs 22 / 22