Graphical Models Reconstruction

Transcription

1 Graphical Models Reconstruction Graph Theory Course Project Firoozeh Sepehr April 27 th 2016 Firoozeh Sepehr Graphical Models Reconstruction 1/50

2 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 2/50

3 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 3/50

4 Overview What are graphical models? Graphical Models 1;2 Combination of Probability Theory and Graph Theory Tackling problems of uncertainty and complexity Utilizing modularity for complex systems Graphical representation of dependencies embedded in probabilistic models b b c c a a g g e e d d f f Bayesian/Belief Networks Markov Networks Firoozeh Sepehr Graphical Models Reconstruction 4/50

5 Overview Markov vs Bayesian Networks Markov Networks Undirected graphical models Correlations between variables Mostly used in physics and vision communities Bayesian/Belief Networks Directed graphical models Directed Acyclic Graphs (DAGs) Causal relationships between variables Mostly used in AI and machine learning communities Use Bayes rule for inference Firoozeh Sepehr Graphical Models Reconstruction 5/50

6 Overview Applications So many different applications Pattern recognition Diagnosis of diseases Desicion-theoretic systems 4 Statistical physics Signal and image processing Inferring cellular networks in biological systems 3 Firoozeh Sepehr Graphical Models Reconstruction 6/50

7 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 7/50

8 History and Background Probability theory Foundations of probability theory 2 Go back to 16th century when Gerolamo Cardano began a formal analysis of games of chance, followed by additional key developments by Pierre de Fermat and Blaise Pascal in 17th century. The initial development involved only discrete probability spaces and the analysis methods were purely combinatorial. Gerolamo Cardano Pierre de Fermat Blaise Pascal Italian, French, French, Science, maths, Mathematics and law 10 Theology, mathematics, philosophy, and literature 9 philosophy and physics 11 Firoozeh Sepehr Graphical Models Reconstruction 8/50

9 History and Background Probability theory Foundations of probability theory - cont d The foundations of modern probability theory were laid by Andrey Kolmogorov in the 1930s. Andrey Kolmogorov Russian, Mathematics Known for Topology, Intuitionistic logic, Turbulence studies, Classical mechanics, Mathematical analysis, Kolmogorov complexity 12 Firoozeh Sepehr Graphical Models Reconstruction 9/50

10 History and Background Bayes rule Bayes theorem 2 Shown in the 18th century by Reverend Thomas Bayes. This theorem allows us to use a model that tells us the conditional probability of event a given event b in order to compute the contrapositive: the conditional probability of event b given event a. This type of reasoning is central to the use of graphical models - Bayesian network. Thomas Bayes English, Statistician, philosopher and Presbyterian minister 13 Firoozeh Sepehr Graphical Models Reconstruction 10/50

11 History and Background Origins of graphical models Origins of graphical models 2 Representing interactions between variables in a multidimensional distribution using a graph structure originates in several communities Statistical physics: Gibbs - used an undirected graph to represent the distribution over a system of interacting particles Genetics: path analysis of Sewal Wright - proposed the use of a directed graph to study inheritance in natural species Statistics: Bartlett - analyzing interactions between variables in the study of contingency tables, also known as log-linear models Computer science: Artificial Intelligence (AI) to perform difficult tasks such as oil-well location or medical diagnosis, at an expert level Firoozeh Sepehr Graphical Models Reconstruction 11/50

12 History and Background Origins of graphical models Expert systems 2 Need for methods that allow the interation of multiple pieces of evidence and provide support for making decisions under uncertainty Huge success in predicting the diseases using evidences like sysmptoms and test results in the 1970s Fell into disfavor in AI community 1 AI should be based on similar methods to human intelligence 2 Use of strong independence assumptions mae in the existing expert systems was not a flexible, scalable mechanism Firoozeh Sepehr Graphical Models Reconstruction 12/50

13 History and Background Origins of graphical models Expert systems - cont d Widespread acceptance of probabilistic methods began in the late 1980s 1 Series of seminal theoretical developments Bayesian network framework by Judea Pearl and his colleaagues in 1988 Foundations for efficient reasoning using probabilistic graphical models by S. L. Lauritzen and D.J. Spiegelhalter in Construction of large-scale, highly successful expert systems based on this framework that avoided the unrealistically strong assumptions made by early probabilistic expert systems Pathfinder expert system (which assists community pathologists with the diagnosis of lymph-node pathology) constructed by Heckerman and colleagues in Firoozeh Sepehr Graphical Models Reconstruction 13/50

14 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 14/50

15 Graphical Models Definitions Directed and undirected graphs G = (N, E) is an undirected graph G = (N, E) a directed graph Degree, indegree and outdegree For a vertex y N degree is deg(y) indegree is deg (y) outdegree is deg + (y) Root and leaf If deg (y) = 0, y is a root and if deg + (y) = 0, y is a leaf Firoozeh Sepehr Graphical Models Reconstruction 15/50

16 Graphical Models Definitions Chains and paths A chain starting from y i and ending in y j is an ordered sequence of distinct nodes (y π1, y π2,..., y πl 1, y πl ) where y i = y π1, y j = y πl and (y k, y k+1 ) E A path starting from y i and ending in y j is an ordered sequence of distinct nodes (y π1, y π2,..., y πl 1, y πl ) where y i = y π1, y j = y πl and either (y k, y k+1 ) E or (y k+1, y k ) E Note Chains are a special case of paths! Firoozeh Sepehr Graphical Models Reconstruction 16/50

17 Graphical Models Definitions Parents, Children, Ancestors, Descendants Consider a directed graph G = (N, E) and y i N. Given a set X N: y i is a parent of y j if there is a directed edge from y i to y j pa(x ) := {y i N y j X : y i is a parent of y j } y j is a child of y i if there is a directed edge from y i to y j ch(x ) := {y j N y i X : y j is a child of y i } y i is an ancestor of y j if there is a chain from y i to y j an(x ) := {y i N y j X : y i is an ancestor of y j } y j is a descendant of y i if there is a chain from y i to y j de(x ) := {y j N y i X : y j is a descendant of y i } Neighbors ngb(y i ), are the union of parents and children set. Firoozeh Sepehr Graphical Models Reconstruction 17/50

18 Graphical Models Definitions Visualize... Roots, Leaves Paths, Chains Parents, Children, Ancestors, Descendants, Neighbors a d g b f e c Firoozeh Sepehr Graphical Models Reconstruction 18/50

19 Graphical Models Definitions Forks, inverted forks and chain links 6 Consider a path (y π1, y π2,..., y πl 1, y πl ) in a directed graph G = (N, E). Vertex y πi is a fork if (y πi, y πi 1 ) and (y πi, y πi+1 ) are in E an inverted fork (or collider) if (y πi 1, y πi ) and (y πi+1, y πi ) are in E a chain link in all other cases a d g b f e c Firoozeh Sepehr Graphical Models Reconstruction 19/50

20 Graphical Models What is factorization? Factorization Joint probability distribution Using the chain rule and assuming an arbitrary order d on variables 2 p(x 1, x 2,..., x n ) = Π n i=1p(x i x 1, x 2,..., x i 1 ) (1) Using graphical models - leads to a compact representation 8 Undirected GM p(x 1, x 2,..., x n) = 1 Z Π (i,j) E φ k (x i, x j ) (2) Undirected Tree GM (using junction tree theory) p(x 1, x 2,..., x n) = Π n i=1p(x i )Π (i,j) E p(x i, x j ) p(x i )p(x j ) (3) Directed GM p(x 1, x 2,..., x n) = Π n i=1p(x i pa(x i )) (4) Firoozeh Sepehr Graphical Models Reconstruction 20/50

21 Graphical Models What is factorization? Example 1 Consider we have N binary random variables, for representation of joint probability distribution chain rule requires O(2 N ) parameters GM requires O(2 pa ) which could reduce the number of parameters exponentially depending on which conditional assumptions we make - helps in inference and learning Firoozeh Sepehr Graphical Models Reconstruction 21/50

22 Graphical Models What is factorization? Example 2 Joint probability distribution 1 Using the chain rule p(x 1, x 2, x 3, x 4, x 5, x 6 ) = p(x 1 )p(x 2 x 1 )p(x 3 x 1, x 2 ) p(x 4 x 1, x 2, x 3 )p(x 5 x 1, x 2, x 3, x 4 ) p(x 6 x 1, x 2, x 3, x 4, x 5 ) Using graphical models p(x 1, x 2, x 3, x 4, x 5, x 6 ) = p(x 1 )p(x 2 x 1 )p(x 3 x 2, x 5 ) p(x 4 x 1 )p(x 5 x 4 )p(x 6 x 5 ) (5) (6) x 1 x 2 x 3 x 6 x 4 x 5 Firoozeh Sepehr Graphical Models Reconstruction 22/50

23 Graphical Models Fun application of joint distribution factorization In rooted trees Joint probability distribution is the same! Use the Bayes rule... b b b b a c a c a c a Undirected a is root b is root c is root c p(a, b, c) = p(a)p(b a)p(c b) = p(a) p(b) p(a, b) p(b) p(a) p(c b) = p(b)p(a b)p(c b) = p(c)p(b c)p(a b) (7) Firoozeh Sepehr Graphical Models Reconstruction 23/50

24 Graphical Models Undirected Graphical Models Undirected graphical models Family of multivariate probability distributions that factorize according to a graph G = (N, E) Set of vertices, N, represents random variables Set of edges, E, encodes the set of conditional independencies between variables Definition Random vector X is said to be Markov on G if for every i, the random variable x i is conditionally independent of all other variables given its neighbours. p(x i x \i ) = p(x i ngb(x i )) (8) where p is the joint probability distribution. Firoozeh Sepehr Graphical Models Reconstruction 24/50

25 Graphical Models Undirected Graphical Models Tree-structured graphical models Family of multivariate probability distributions that are Markov on a tree T = (N, E) Firoozeh Sepehr Graphical Models Reconstruction 25/50

26 Graphical Models Definition d-separation 6 A subset of variables S is said to separate x i from x j if all paths between x i and x j are separated by S A path P is separated by a subset S of variables if at least one pair of successive edges along P is blocked by S block 6 Two edges meeting head-to-tail or tail-to-tail at node x (x is a chain or a fork) are blocked by S if x is in S Two edges meeting head-to-head at node x (x is an inverted fork) are blocked by S if neither x nor any of its descendants is in S. Firoozeh Sepehr Graphical Models Reconstruction 26/50

27 Graphical Models Definition d-separation Example 6 d sep(x 2, x 3 {x 1 })? d sep(x 2, x 3 {x 1, x 4 })? d sep(x 2, x 3 {x 1, x 6 })? x 4 x 2 x 1 x 3 x 5 x 6 Firoozeh Sepehr Graphical Models Reconstruction 27/50

28 Graphical Models Interesting application Lumiere project 5 The Lumiere Project centers on harnessing probability and utility to provide assistance to computer software users. Lumiere prototypes served as the basis for components of the Office Assistant in the Microsoft Office 97 suite of productivity applications. Infers a user s needs by considering a user s background, actions, and queries Challenges are Model construction about time-varying goals of computer users Needs a large database - over 25,000 hours of usability studies were invested in Office 97 Firoozeh Sepehr Graphical Models Reconstruction 28/50

29 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 29/50

30 Reconstruction What is reconstruction? Reconstruction The problem is that samples are available only from a subset of variables The goal is to learn the minimal latent tree - trees without any redundant hidden nodes Latent and minimal latent trees Note A latent tree is a tree with node set N = V H, where V is the set of observed nodes and H is the set of latent (hidden) nodes. Set of minimal latent trees, T 3, is the set of latent trees that each hidden node has at least three neighbors (hidden or observed) All leaves are observed, although not all observed nodes need to be leaves. Firoozeh Sepehr Graphical Models Reconstruction 30/50

31 Graphical Models Interesting application Vista system 4 A decision-theoretic system that has been used at NASA Mission Control Center in Houston for several years. Uses Bayesian networks to interpret live telemetry and provides advice on the likelihood of alternative failures of the space shuttle s propulsion systems. Considers time criticality and recommends actions of the highest expected utility Employs decision-theoretic methods for controlling the display of information to dynamically identify the most important information to highlight Firoozeh Sepehr Graphical Models Reconstruction 31/50

32 Reconstruction Additive metric Define a measurement 8 Information distances Defined for pairwisse distributions For guassian graphical models, correlation coefficient of two random variables x i and x j ρ ij = cov(x i, x j ) var(xi )var(x j ) (9) Information distance d ij = log ρ ij (10) Inverse relation between information distance and correlation Extendable to discrete random variables Firoozeh Sepehr Graphical Models Reconstruction 32/50

33 Reconstruction Additive metric Proposition 8 The information distances d ij are additive tree metrics. In other words, if the joint probabiliry distribution p(x) is a tree-structured graphical model Markov on the tree T p = (N, E p ), then the information distances are additive on T p. k, l N : d kl = (i,j) Path kl d ij (11) Proof Homework! Firoozeh Sepehr Graphical Models Reconstruction 33/50

34 Reconstruction Sibling grouping Lemma 8 For distances d ij for all i, j V on a tree T T 3, the following two properties on Φ ijk = d ik d jk hold. 1 Φ ijk = d ij for all k V \i,j iff i is a leaf and j is its parent 1 Φ ijk = d ij for all k V \i,j iff j is a leaf and i is its parent 2 d ij < Φ ijk = Φ ijk < d ij for all k, k V \i,j iff both i and j are leaves and they have the same parent (they belong to the same sibling group) Proof of 2 Homework! Firoozeh Sepehr Graphical Models Reconstruction 34/50

35 Reconstruction Sibling grouping Proof of 1 : Using the additive property of information distances, if i is a leaf and j is its parent, d ik = d ij + d jk, therefore, Φ ijk = d ij for all k i, j. : By contradiction, i and j are not connected with an edge. Then there exists a node u i, j on the path connecting i and j. If u V, then let k = u, otherwise, let k be an observed node in the subtree away from i and j which exists since T T 3. Therefore, d ij = d iu + d uj > d iu d uj = d ik d kj = Φ ijk which is a contradiction. i u j k Firoozeh Sepehr Graphical Models Reconstruction 35/50

36 Reconstruction Sibling grouping Proof of 1 - cont d : By contradition, if i is not a leaf, then there exists a node u i, j such that (i, u) E. Let k = u if u V, otherwise, let k be an observed node in the subtree away from i and j. Therefore, Φ ijk = d ik d jk = d ij < d ij which is again a contradiction, therefore, i is a leaf. j i u k Firoozeh Sepehr Graphical Models Reconstruction 36/50

37 Reconstruction Sibling grouping Using previous Lemma to determine node relationships 8 For every pair of i, j V consider the following: 1 If Φ ijk = d ij for all k V \i,j, then i is a leaf node and j is a parent of i. Similarly, if Φ ijk = d ij for all k V \i,j, then j is a leaf and i is a parent of j. 2 If Φ ijk is constant for all k V \i,j but not equal to either d ij or d ij, then i and j are leaves and they are siblings. 3 If Φ ijk is not equal for all k V \i,j, then there are three cases: (a) Nodes i and j are not siblings nor have a parent-child relationship. (b) Nodes i and j are siblings but at least one of them is not a leaf. (c) Nodes i and j have a parent-child relationship but the child is not a leaf. Firoozeh Sepehr Graphical Models Reconstruction 37/50

38 Reconstruction Sibling grouping Visualize... Case 1 Case 2 d1 d1 d3 d3 d2 d2 j d4 d8 i d5 d6 d7 d4 d8 d5 d6 i d7 j Φ ijk = d 8 = d ij Φ ijk d ij Φ ijk = d 6 d 7 d ij = d 6 + d 7 for all k V \ i, j Firoozeh Sepehr Graphical Models Reconstruction 38/50

39 Reconstruction Sibling grouping Visualize... Case 3a Case 3b Case 3c d1 d1 d1 i d4 k d2 d5 d6 d3 k d7 j i d4 d2 k j d5 d6 d3 d7 k d4 i j d2 d5 d6 d3 d7 d8 k d8 k d8 Φ ijk Φ ijk Φ ijk Φ ijk Φ ijk Φ ijk Φ ijk = d 4 + d 2 + d 3 d 7 Φ ijk = d 4 + d 5 Φ ijk = d 4 d 5 Φ ijk = d 4 d 2 d 3 d 7 Φ ijk = d 5 Φ ijk = d 5 for all k, k V \ i, j Firoozeh Sepehr Graphical Models Reconstruction 39/50

40 Reconstruction Recursive Grouping (RG) Algorithm Recursive Grouping (RG) Algorithm 1 Initialize Y = V 2 Compute Φ ijk = d ik d jk for all i, j, k Y 3 Using sibling grouping, define {Π l } L l=1 to be partitions of Y such that for every subset Π l (with Π l 2), any two nodes are either siblings which are leaves or they have a parent-child relationship in which the child is a leaf 4 Add singles sets to Y new 5 For each Π l with Π l 2, if Π l contains a parent node, add it to Y new, otherwise, create a new hidden node and connect it to all the nodes in Π l and add the node to Y new 6 Update Y old to be Y and Y to be Y new 7 Compute the distances of new hidden nodes 8 If Y 3, go to step 2, otherwise, if Y = 2, connect two remaining nodes in Y and stop. If Y = 1, stop. Firoozeh Sepehr Graphical Models Reconstruction 40/50

41 Reconstruction Recursive Grouping (RG) Algorithm Visualize... h 2 h Original latent tree h h 1 First iteration 3 h 2 h 3 h 2 h h Second iteration Third iteration h 1 3 Firoozeh Sepehr Graphical Models Reconstruction 41/50

42 Reconstruction Recursive Grouping (RG) Algorithm Proof of step 7 7 Compute the distances of new hidden nodes Let i, j ch(h) and k Y old i, j. We know that d ih d jh = d ik d jk = Φ ijk and d ih + d jh = d ij. Therefore, we can recover the distances between a previously active node i Y old and its new hidden parent h Y using d ih = 1 2 (d ij + Φ ijk ) (12) For any other active node l Y, we can compute d hl using a child node i ch(h) using { dil d d hl = ih, if l Y old d ik d ih d lk, otherwise, where k ch(l) (13) Firoozeh Sepehr Graphical Models Reconstruction 42/50

43 Reconstruction Recap Steps to learn a latent tree 1 Define an additive metric 2 Perform sibling grouping test to determine nodes relationships 3 Perform RG algorithm Firoozeh Sepehr Graphical Models Reconstruction 43/50

44 Outline 1 Overview 2 History and Background 3 Graphical Models 4 Reconstruction 5 Open Issues Firoozeh Sepehr Graphical Models Reconstruction 44/50

45 Open Issues What next? Improvement! Probabilistic models are used as a key component in some challenging applications and they remain to be applied in some other fields Learning other types of GMs Polytrees General graphs Applying the theorems on random processes Define interrelations Firoozeh Sepehr Graphical Models Reconstruction 45/50

46 Homework Question 1 Prove that information distances are additive tree metrics. Question 2 Prove that for distances d ij for all i, j V on a tree T T 3, the following the following property on Φ ijk = d ik d jk holds 2 d ij < Φ ijk = Φ ijk < d ij for all k, k V \i,j iff both i and j are leaves and they have the same parent (they belong to the same sibling group) Firoozeh Sepehr Graphical Models Reconstruction 46/50

47 Homework Question 3 Draw the digraph associated with the following matrix and answer the followings. d sep(x 1, x 2 {x 6, x 7})? d sep(x 4, x 5 {x 1, x 2, x 3, x 6})? d sep(x 1, x 7 {x 3, x 4, x 5})? M = (14) Firoozeh Sepehr Graphical Models Reconstruction 47/50

48 Questions? Firoozeh Sepehr Graphical Models Reconstruction 48/50

49 References I [1] Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference J. Pearl, 1988 [2] Probabilistic Graphical Models, Principles and Techniques D. Koller, N. Friedman, MIT Press, 2009 [3] Inferring Cellular Networks Using Probabilistic Graphical Models N. Friedman, Vol 303, Issue 5659, pp , 2004 [4] Vista Goes Online: Decision-Analytic Systems for Real-Time Decision-Making in Mission Control M. Barry, E. Horvitz, C. Ruokangas, S. Srinivas, N , 1994 [5] The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users E. Horvitz, J. Breese, D. Heckerman, D. Hovel, K. Rommelse, 1998 [6] Fusion, Propagation, and Structuring in Belief Networks J. Pearl, Artificial Intelligence 29, 1986 Firoozeh Sepehr Graphical Models Reconstruction 49/50

50 References II [7] The Recovery of Causal Polytrees from Statistical Data G. Rebane, J. Pearl, Proceedings of the Third Conference on Uncertainty in Artificial Intelligence, 1987 [8] Learning Latent Tree Graphical Models M. J. Choi, V. Y. F. Tan, A. S. Willsky, Journal of Machine Learning Research, Volume 12, 2011 [9] Gerolamo Cardano Cardano [10] Pierre de Fermat de Fermat [11] Blaise Pascal Pascal [12] Andrey Kolmogorov Kolmogorov [13] Thomas Bayes Bayes [14] An Evaluation of the Diagnostic Accuracy of Pathfinder D. E. Heckerman, 1991 Firoozeh Sepehr Graphical Models Reconstruction 50/50