2. Graphical Models. Undirected graphical models. Factor graphs. Bayesian networks. Conversion between graphical models. Graphical Models 2-1

Transcription

1 Graphical Models Graphical Models Undirected graphical models Factor graphs Bayesian networks Conversion between graphical models

2 Graphical Models 2-2 Graphical models There are three families of graphical models that are closely related, but suitable for different applications and different probability distributions: Undirected graphical models (also known as Markov Random Fields) Factor graphs Bayesian networks we will learn what they are, how they are different and how to switch between them. consider a probability distribution over x = (x 1, x 2,..., x n ) µ(x 1, x 2,..., x n ) a graphical model is a graph and a set of functions over a subset of random variables which define the probability distribution of interest

3 graphical model is a marriage between probability theory and graph theory that allows compact representation and efficient inference, when the probability distribution of interest has special independence and conditional independence structures for example, consider a random vector x = (x 1, x 2, x 3 ) and a given distribution µ(x 1, x 2, x 3 ) we use (with a slight abuse of notations) µ(x 1 ) x 2,x 3 µ(x 1, x 2, x 3 ), and µ(x 1, x 2 ) x 3 µ(x 1, x 2, x 3 ) to denote the first order and the second order marginals respectively we can list all possible independence structures x 1 (x 2, x 3 ) µ(x 1, x 2, x 3 ) = µ(x 1 )µ(x 2, x 3 ) (1) x 1 x 2 µ(x 1, x 2 ) = µ(x 1 )µ(x 2 ) (2) x 1 x 2 x 3 x 1 x 3 x 2 µ(x 1, x 2 x 3 ) = µ(x 1 x 3 )µ(x 2 x 3 )(3) and various permutations and combinations of these Graphical Models 2-3

4 Graphical Models 2-4 warm-up exercise (1) (2) proof: µ(x 1, x 2 ) = x 3 µ(x 1, x 2, x 3 ) (1) = x 3 µ(x 1 )µ(x 2, x 3 ) = µ(x 1 )µ(x 2 ) (2) (3) counter example: X 1 X 2 and X 3 = X 1 + X 2 (2) (3) counter example: Z 1, Z 2, X 3 are independent and X 1 = X 3 + Z 1, X 2 = X 3 + Z 2 this hints that different graphical models are required to represent different notions of independence in fact, there are exponentially many possible independencies, resulting in doubly exponentially many possible independence structures in a distribution however, there are only 2 n2 undirected graphs (4 n2 for directed) hence, graphical models only capture (important) subsets of possible independence structures

5 a probabilistic graphical model is a graph G(V, E) representing a family of probability distributions 1. that share the same factorization of the probability distribution; and 2. that share the same independence structure. undirected graphical model = Markov Random Field (MRF) µ(x) = 1 ψ c (x c ) Z c C(G) where C(G) is the set of all maximal cliques in the undirected graph G(V, E) factor graph model (FG) µ(x) = 1 ψ a (x a ) Z a F where F is the set of factor nodes in the undirected bipartite graph G(V, F, E) and a is the set of neighbors of the node a directed graphical model = Bayesian Network (BN) µ(x) = i V µ(x i x π(i) ) where π(i) is the set of parent nodes in the directed graph G(V, E) Graphical Models 2-5

6 Graphical Models 2-6 warm-up example: Markov Random Fields (MRF) and Factor Graphs (FG) MRF FG factorization independence x 1 x 2 x 3 µ(x 1, x 2, x 3 ) none ψ(x 1, x 2 )ψ(x 1, x 3 ) x 2 x 3 x 1 [HW1.1] ψ(x 1, x 2 )ψ(x 3 ) x 3 (x 1, x 2 ) ψ(x 1 )ψ(x 2 )ψ(x 3 ) all indep.

7 Graphical Models 2-7 warm-up example: Bayesian Network (BN) of ordering (x 1 x 2 x 3 ) BN x 1 x 2 factorization µ(x 1 )µ(x 2 x 1 )µ(x 3 x 1, x 2 ) independence none x 3 µ(x 1 )µ(x 2 x 1 )µ(x 3 x 1 ) x 2 x 3 x 1 µ(x 1 )µ(x 2 )µ(x 3 x 1, x 2 ) x 1 x 2 µ(x 1 )µ(x 2 x 1 )µ(x 3 x 2 ) x 1 x 3 x 2 µ(x 1 )µ(x 2 x 1 )µ(x 3 ) x 3 (x 1, x 2 ) µ(x 1 )µ(x 2 )µ(x 3 ) all indep.

8 Family #1: Undirected Pairwise Graphical Models Graphical Models 2-8

9 Graphical Models 2-9 Family #1: Undirected Pairwise Graphical Models (a.k.a. Pairwise MRF) x 10 x 2 x 3 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 x 6 G = (V, E), V = [n] {1,..., n}, x = (x 1,..., x n ), x i X

10 x 2 x 3 x 10 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 Undirected pairwise graphical models are specified by Graph G = (V, E) Alphabet X Compatibility function ψ ij : X X R +, for all (i, j) E µ(x) = 1 Z (i,j) E x 6 ψ ij (x i, x j ) pairwise MRF only allow compatibility functions over two variables Graphical Models 2-10

11 Graphical Models 2-11 Undirected Pairwise Graphical Models Alphabet X Typically X < Occationally X = R and µ(dx) = 1 Z ψ ij (x i, x j ) dx (i,j) E (all formulae interpreted as densities) Compatibility function ψ ij µ(x) = 1 Z ψ ij (x i, x j ) (i,j) E Partition function Z plays a crucial role! Z = ψ ij (x i, x j ) x X V (i,j) E

12 Graph notation i {neighborhood of vertex i}, 9 = {5, 6, 7} deg(i) = i, deg(9) = 3 x U (x i ) i U, x {1,5} = (x 1, x 5 ) x 9 = (x 5, x 6, x 7 ) Complete graph Clique Graphical Models 2-12

13 Example x 2 x 3 x 10 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 Coloring (e.g. ring tone) Given graph G = (V, E) and a set of colors X = {R, G, B} Find a coloring of the vertices such that no two adjacent vertices have the same color Fundamental question: Chromatic number Graphical Models 2-13 x 6

14 x 2 x 3 x 10 x 11 x 1 x 4 x 12 x 9 x 5 x 7 A (joint) probability of interest is uniform measure over all possible colorings: µ(x) = 1 I(x i x j ) Z x8 (i,j) E Z = total number of colorings Sampling = coloring similarly, independent set problem [HW1.3, 1.4] Graphical Models 2-14 x 6

15 (General) Undirected Graphical Model x 10 x 2 x 3 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 x 6 Undirected graphical models are specified by Graph G = (V, E) Alphabet X Compatibility function ψ c : X c R +, for all maximal cliques c C µ(x) = 1 ψ c (x c ) Z Graphical Models 2-15 c C

16 Family #2: Factor Graph Models Graphical Models 2-16

17 Family #2: Factor graph models 4 b x 4 c x 5 x 6 variable x i X x 3 d x 2 x 1 factor ψ a (x 1, x 5, x 6 ) a }{{} a Factor graph G = (V, F, E) Variable nodes i, j, k, V Function nodes a, b, c, F Variable node x i X, for all i V Function node ψ a : X a R +, for all a F µ(x) = 1 ψ a (x a ) Z Graphical Models 2-17 a F

18 Factor graph models 4 b x 4 c x 5 x 6 variable x i X x 3 d x 2 x 1 factor ψ a (x 1, x 5, x 6 ) a }{{} a Factor graph model is specified by Factor graph G = (V, F, E) Alphabet X Compatibility function ψ a : X a R +, for a F µ(x) = 1 ψ a (x a ) Z a F Partition function: Z = x X a F V ψ a(x a ) Graphical Models 2-18

19 Conversion between factor graphs and pairwise models From pairwise model to factor graph A pairwise model on G(V, E) with alphabet X can be represented by a factor graph G (V, F, E ) with V = V, F E, E = 2 E, X = X. Put a factor node on each edge From factor graph to a general undirected graphical model A factor model on G(V, F, E) with alphabet X can be represented by a pairwise model on G (V, E ) with V = V, E a F a 2, X = X. A factor node is turned into a clique From factor graph to a pairwise model A factor model on G(V, F, E) can be represented by a pairwise model on G (V, E ) with V = V F, E = E, X = X, = max a F deg(a). A factor node is represented by a large variable node Graphical Models 2-19

20 Family #3: Bayesian Networks Graphical Models 2-20

21 Graphical Models 2-21 Family #3: Bayesian networks x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 DAG: Directed Acyclic Graph G = (V, D) Variable nodes V = [n], x i X, for all i V Define π(i) {parents of i} Set of directed edges D µ(x) = i V µ i (x i x π(i) )

22 Graphical Models 2-22 x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Bayesian network is specified by directed acyclic graph G = (V, D) alphabet X conditional probability µi ( ) : X X π(i) R +, for i V µ(x) = i V µ i (x i x π(i) ) we do not need normalization (1/Z) since µ i (x i x π(i) ) = 1 x i X x X V µ(x) = 1

23 Conversion between Bayes networks and factor graphs from Bayes network to factor graph A Bayes network G = (V, D) with alphabet X can be represented by a factor graph model on G = (V, F, E ) with V = V, F = V, E = D + V, X = X. represent by a factor node each conditional probability moralization for conversion from BN to MRF from factor graph to Bayes network A factor model on G = (V, F, E) with alphabet X can be represented by a Bayes network G = (V, D ) with V = V and X = X. take a topological ordering, e.g. x 1,..., x n for each node i, starting from the first node, find a minimal set U {1,..., i 1} such that x i is conditionally independent of x {1,...,i 1}\U given x U. (we will learn how to do this) in general the resulting Bayesian network is dense Graphical Models 2-23

24 Because MRF and BN are incomparable, some independence structure is lost in conversion Graphical Models 2-24 µ(x) = ψ(x 1, x 2)ψ(x 1, x 3)ψ(x 2, x 4)ψ(x 3, x 4) x 1 x 4 (x 2, x 3) x 2 x 3 (x 1, x 4) x 2 x 3 (x 1, x 4) µ(x) = µ(x 2)µ(x 3)µ(x 1 x 2, x 3) x 2 x 3 no independence

25 Graphical Models 2-25 Factor graphs are more fine grained than undirected graphical models ψ(x 1, x 2, x 3 ) ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 )ψ 31 (x 3, x 1 ) ψ 123 (x 1, x 2, x 3 ) all three encodes same independencies, but different factorizations (in particular the degrees of freedom in the compatibility functions are 3 X 2 vs. X 3 ) set of independencies represented by MRF is the same as FG but FG can represent a larger set of factorizations

26 Graphical Models 2-26 undirected graphical models can be represented by factor graphs we can go from MRF to FG without losing any information on the independencies implies by the model Bayesian networks are not compatible with undirected graphical models or factor graphs if we go from one model to the other, and then back to the original model, then we will not, in general, get back the same model as we started out with we lose any information on the independencies implies by the model, when switching from one model to the other

27 Graphical Models 2-27 Bayes networks with observed variables V = H O Hidden variables: x = (x i ) i H Observed variables: y = (y i ) i O µ(x, y) = i H µ(x i x π(i) H, y π(i) O ) i O µ(y i x π(i) H, y π(i) O ) Typically interested in µ y (x) µ(x y) and arg max x µ y (x)

28 Graphical Models 2-28 Example Forensic Science [Kadane, Shum, A probabilistic analysis of the Sacco and Vanzetti evidence, 1996] [Taroni et al., Bayesian Networks and Probabilistic Inference in Forensic Science, 2006]

29 Graphical Models 2-29 Example diseases soft ORs symptoms Medical Diagnosis [M. Shwe, et al., Methods of Information in Medicine, 1991]

30 Graphical Models 2-30 Roadmap Cond. Indep. Factorization Graphical Graph Cond. Indep. µ(x) µ(x) Model G implied by G 1 x 1 {x 2, x 3 } x 4 ; Z ψa (x a ) FG Factor Markov 1 x 4 {} x 7 ; Z ψc (x C ) MRF Undirected Markov. ψi (x i x π(i) ) BN Directed Markov Any µ(x) can be represented by {FG,MRF,BN} A µ(x) can be represented by multiple {FG,MRF,BN} with multiple graphs (but same µ(x)) We want a simple graph representation (sparse, small alphabet size) Memory to store the graphical model Computations for inference µ(x) with some conditional independence structure can be represented by simple {FG,MRF,BN}