Learning decomposable models with a bounded clique size

Transcription

1 Learning decomposable models with a bounded clique size Achievements Aritz Pérez Basque Center for Applied Mathematics Bilbao, March, 2016

2 Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

3 Motivation and background Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

4 Motivation and background Decomposable models (DM) Probability distributions Probabilistic graphical models (PGM) based on decomposable graphs. The intersection between Markov networks and Bayesian networks. Advantageous properties: Maximum likelihood parameters in closed form. Graphically described dependence models. Closed product-form expression in term of marginal probability distributions. Closed form expressions for test statistics. The computational complexity for performing inference is bounded by the maximum clique size. Goal Structural learning of decomposable models with a bounded clique size.

5 Motivation and background Learning decomposable models Structural learning using data of decomposable models with a bounded clique size by maximizing additively decomposable scores. Efficiency: deal with big graphs Quality: obtain models good quality

6 Motivation and background Undirected graphs Definition G = (V, E) is an undirected graph, where V = {1,..., n} is a set of indexes called vertex, and E is a set of pair of vertices called edges.

7 Motivation and background Coarser and thinner Definition We say that G + = (V, E + ) is coarser than G = (V, E) (and G is thinner than G + ), when E E +, and it is denoted by G + G.

8 Motivation and background Decomposable graphs Definition An undirected graph is said to be decomposable (chordal,...) when every cicle of length higher than three has a chorde.

9 Motivation and background Cliques and separators Definitions Clique: the vertices of a maximal complete subgraph. Separator: a minimal set that blocks all the paths from u to v.

10 Motivation and background Candidate edge Definition Let G be a decomposable graph. We say that an edge is a candidate edge for G if its addition to G produces a decomposable graph.

11 Motivation and background Candidate edge Characterization Given a decomposable graph G = (V, E), an edge {u, v} / E is a candidate edge if and only if there exists a separator S for u and v, such that {u, v} N(S). The addition of {u, v} to G creates the clique {u, v} S.

12 Motivation and background k-order decomposable graphs Definition A k-order decomposable graph (kdg) is a decomposable graph for which the maximum clique size is k. A maximal k-order decomposable graph (MkDG) is a kdg for which all the cliques are of size k and the addition of a candidate edge creates a clique of size k + 1.

13 The problem Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

14 The problem Additively decomposable scores Definition w(g) = R C(G + ), where w R R and C(G + ) is the set of all complete sets of G. Decomposable scores for PGMs Log likelihood, Bayesian Dirichlet equivalent score, minimum description length,... w R

15 The problem Problem kdg Definition Learn the kdg that maximizes the decomposable score w(g). Problem kdg is NP-hard for k > 2.

16 A divide-and-conquer strategy Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

17 A divide-and-conquer strategy A divide-and-conquer strategy Intuition Construct a sequence of idgs G i for i = 1,..., k in k 1 growing steps, where G 1 G 2 G 3... G k 1 G k

18 A divide-and-conquer strategy Problem (k + 1)DG Definition Given an MkDG, learn a coarser (k + 1)DG that maximizes the score w(g). Problem (k + 1)DG is still NP-hard for k > 2.

19 A divide-and-conquer strategy Properties behind Problem (k + 1)DG Problem (k + 1)DG can be solved by adding a sequence of candidate edges to the given MkDG. The set of candidate edges that create cliques of size (k + 1) can be identified by a local inspection of the separators of size k 1. All the separators of an MkDG are of size k 1. The set of separators of size k 1 for a (k + 1)DG is a subset of the set of separators of any thinner MkDG.

20 A divide-and-conquer strategy Proposals for dealing with Problem (k + 1)DG Efficient approach. Exact approach. Flexible approach.

21 An efficient approach Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

22 An efficient approach Fractal tree Learning the maximum log. likelihood decomposable models with a bounded clique size. Log. likelihood is a monotone increasing function with respect to the edges added. The solutions are MK DGs. Fractal trees have O(k 2 n 2 N) computational complexity and they can be easily parallelized.

23 An efficient approach Separator-based decomposition Mantle of a separator The subgraph associated to the common neighborhood of a separator. Decomposes an MkDG (or a coarser (k + 1)DG) into forests.

24 An efficient approach Separator problem Definition Given a mantle learn a coarser tree that maximizes the log. likelihood. This problem can be solved in O(k m 2 N), where m is the number of vertices in the mantle. The partial solutions to each separator problem can be gathered together to form an M(k + 1)DG.

25 An efficient approach The fractal tree family Parallel Ignores the interactions between the mantles. Solve all separator problems in parallel. Sequential fractal tree Take into account the interactions between the mantles. Solve the separator problems sequentially, in a given order, taking into account their interactions. It considers more candidate edges than the parallel version.

26 An efficient approach Prune & Graft procedure Given an MkDG, P&G obtains an MkDG with equal or higher log likelihood. It favors the mobility of the vertices between the mantles of different separators. It is applied at the end of each growing step of the fractal tree algorithms. It has a computational complexity of O(k n 2 N).

27 An efficient approach Empirical results: power of fitting Fitting CL FG PFT SFT True N

28 An efficient approach Empirical results: power of generalization Generalization CL FG PFT SFT True N

29 An efficient approach Empirical results: Time Time (s) CL FG PFT SFT n

30 An efficient approach Conclusions The fractal tree family is very efficient and it can be easily parallelized They obtain competitive results. Next step: Obtain higher quality solutions at expense of a higher computational complexity.

31 An exact approach Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

32 An exact approach What is needed for solving Problem (k + 1)DG? Add all the possible sequences of candidate edges that create cliques of size k + 1 starting from a given MkDG and select the coarser (k + 1)DG that maximizes the decomposable score.

33 An exact approach Edges of interest (EOI) Definition Let G be an MkDG. An edge is called edge of interest (EOI) due to S for G when i) it is a candidate edge for G or for any (k + 1)DG coarser than G and, ii) its addition creates a clique of size k + 1. The set of EOIs due to S for G is given by {{u, v} : u G v S} Select a subset of EOIs that maximize the weight contribution subject to a set of constraints to ensure the decomposability. The number of EOIs is O(n 3 ), in the worst case.

34 An exact approach Integer Linear Programming (ILP) Simulates a sequential addition of candidate edges. Decision variables: EOIs {X u,v S : S S(G), {u, v} E S (G)} X u,v S = { 1, if the EOI {u, v} due to S is included. 0, if the EOI {u, v} due to S is NOT included.

35 An exact approach ILP Function max w u,v S X u,v S S S(G),{u,v} E S (G)

36 An exact approach ILP Constraint 1 For {u, v} E: S S(G) : {u, v} E S (G) X u,v S 1 Each edge can be added once to the solution, due to a single separator

37 An exact approach ILP Constraint 2 For X u,v S : [ [ R S(G) : {u, v} E R (G) R S(G) : {u, v} E R (G) X u,s R ] (k 1) X u,v S s S s S X v,s R ] (k 1) X u,v S s S s S 1 u,s 1 v,s {u, v} is in the neighborhood of S in the solution

38 An exact approach Constraint 2

39 An exact approach ILP Constraint 3 For S S(G) and C comp(s G): X u,v S C 1 T U C u T v U The vertices u and v were separated by S before the edge was added A forest among the connected components induced by each separator is added

40 An exact approach Constraint 3

41 An exact approach Size Worst case Variables: O(n 3 ) Constraint 1: O(n 2 ) Constraint 2: O(n 3 ) Constraint 3: O( S S(G) 2d S )

42 An exact approach Empirical results: Quality of the solution Vars Gap n n APD Edge n n

43 An exact approach Empirical results: Size of the formulation Vars Cons n n Cons Cons n n

44 An exact approach Conclusions An exact ILP formulation of Problem (k + 1)DG. Next step: control the size of the formulation by discarding less promising edges.

45 A flexible approach Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

46 A flexible approach Trade-off effectiveness-efficiency Intuition Reduce the size of the ILP model by discarding edges that, on average, have low probability of being added to the solution.

47 A flexible approach A distance based criteria Threshold distance: l max Reduce the set of EOIs due to S by imposing a constraint based on l m ax : E lmax S (G) = {{u, v} : u G v S and l G (u, v) l max } where l G (u, v) is the length of the edge.

48 A flexible approach properties The selected EOIs are appropriate 1 They can be used to approach Problem (k + 1)DG: the constraints associated to an edge are given in terms of edges of lower length. 2 The selected EOIs have higher probability of being part of the solution to Problem (k + 1)DG than the discarded ones. 3 l max can effectively control the size of the ILP formulation: number of decision variables, constraints (1) and (2), and the number of additive terms involved in the constraints (1), (2) and (3).

49 A flexible approach Empirical results: Quality of the solution Vars Inf time Inf n n Gap APD Inf 0 Inf n n

50 A flexible approach Empirical results: Size of the formulation Vars Inf Cons Inf n n Cons Cons Inf Inf n n

51 A flexible approach Conclusions A distance based criteria that discards less promising edges and controls the size of ILP formulation We can control the trade-off between the quality of the solution and the computational resources required

52 Conclusions and future work Outline 1 Motivation and background 2 The problem 3 A divide-and-conquer strategy 4 An efficient approach 5 An exact approach 6 A flexible approach 7 Conclusions and future work

53 Conclusions and future work Conclusions We have proposed an effective divide-and-conquer strategy to Problem kdg. We have proposed three approaches to Problem (k + 1)DG: an efficient, an exact and a flexible approach.

54 Conclusions and future work Future work Study the ensembles of MkDG by using the efficient approach. Improve the flexible approach by using ILP strategies: linear relaxation, column addition and cutting planes.

55 Conclusions and future work Questions?