2. Graphical Models. Undirected pairwise graphical models. Factor graphs. Bayesian networks. Conversion between graphical models. Graphical Models 2-1

Transcription

1 Graphical Models Graphical Models Undirected pairwise graphical models Factor graphs Bayesian networks Conversion between graphical models

2 Graphical Models 2-2 Graphical models Families of graphical models Undirected pairwise graphical models Factor graphs Bayesian networks Probability distribution over x = (x 1, x 2,..., x n ) µ(x 1, x 2,..., x n )

3 Family #1: Undirected Pairwise Graphical Models Graphical Models 2-3

4 Graphical Models 2-4 Family #1: Undirected Pairwise Graphical Models (a.k.a. Markov Random Fields) x 10 x 2 x 3 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 x 6 G = (V, E), V = [n], x = (x 1,..., x n ), x i X

5 Family #1: Undirected Pairwise Graphical Models x 10 x 2 x 3 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 x 6 Undirected pairwise graphical models are specified by Graph G = (V, E) Alphabet X Compatibility function ψ ij : X X R +, for all (i, j) E µ(x) = 1 Z ψ ij (x i, x j ) (i,j) E Graphical Models 2-5

6 Graphical Models 2-6 Undirected Pairwise Graphical Models Alphabet X Typically X < Occationally X = R and µ(dx) = 1 Z ψ ij (x i, x j ) dx (i,j) E (all formulae interpreted as densities) Compatibility function ψ ij µ(x) = 1 Z ψ ij (x i, x j ) (i,j) E Partition function Z plays a crucial role! Z = ψ ij (x i, x j ) x X V (i,j) E

7 Graph notation i {neighborhood of vertex i}, 9 = {5, 6, 7} deg(i) = i, deg(9) = 3 x U (x i ) i U, x {1,5} = (x 1, x 5 ) x 9 = (x 5, x 6, x 7 ) Complete graph Clique Graphical Models 2-7

8 Example x 2 x 3 x 10 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 Coloring (e.g. ring tone) Given graph G = (V, E) and a set of colors X = {R, G, B} Find a coloring of the vertices such that no two adjacent vertices have the same color Fundamental question: Chromatic number Graphical Models 2-8 x 6

9 Example x 2 x 3 x 10 x 11 x 1 x 4 x 12 x 9 x 5 x 7 A (joint) probability of interest: µ(x) = 1 Z Z = total number of colorings Sampling = coloring x8 (i,j) E x 6 I(x i x j ) Graphical Models 2-9

10 Undirected Graphical Models x 10 x 2 x 3 x 11 x 1 x 4 x 12 x 9 x 5 x 7 x8 x 6 Undirected graphical models are specified by Graph G = (V, E) Alphabet X Compatibility function ψ c : X c R +, for all maximal cliques c C µ(x) = 1 ψ c (x c ) Z Graphical Models 2-10 c C

11 Family #2: Factor Graph Models Graphical Models 2-11

12 Family #2: Factor graph models 4 b x 4 c x 5 x 6 variable x i X x 3 d x 2 x 1 factor ψ a (x 1, x 5, x 6 ) a }{{} a Factor graph G = (V, F, E) Variable nodes i, j, k, V Function nodes a, b, c, F Variable node x i X, for all i V Function node ψ a : X a R +, for all a F µ(x) = 1 ψ a (x a ) Z Graphical Models 2-12 a F

13 Factor graph models 4 b x 4 c x 5 x 6 variable x i X x 3 d x 2 x 1 factor ψ a (x 1, x 5, x 6 ) a }{{} a Factor graph model is specified by Factor graph G = (V, F, E) Alphabet X Compatibility function ψ a : X a R +, for a F µ(x) = 1 ψ a (x a ) Z a F Partition function: Z = x X a F V ψ a(x a ) Graphical Models 2-13

14 Graphical Models 2-14 Conversion between factor graphs and pairwise models From pairwise model to factor graph A pairwise model on G = (V, E) with alphabet X can be represented by a factor graph model on G = (V, F, E ) with V = V, F E, E = 2 E, X = X. Put a factor node on each edge From factor graph to a pairwise model A factor model on G = (V, F, E) with alphabet X can be represented by a pairwise model on G = (V, E ) with V = V F, E = E, X = X, = max a F deg(a). A factor node is represented by a variable node with the state of its neighbors

15 Family #3: Bayesian Networks Graphical Models 2-15

16 Graphical Models 2-16 Family #3: Bayesian networks x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Directed graph G = (V, D) Variable nodes V = [n], x i X, for all i V Define π(i) {parents of i} Set of directed edges D µ(x) = i V µ i (x i x π(i) )

17 Graphical Models 2-17 Bayesian networks x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 Bayesian network is specified by directed acyclic graph G = (V, D) alphabet X conditional probability µi ( ) : X X π(i) R +, for i V µ(x) = µ i (x i x π(i) ) i V µ i (x i x π(i) ) = 1 x i X x X V µ(x)

18 Conversion between Bayes networks and factor graphs from Bayes network to factor graph A Bayes network G = (V, D) with alphabet X can be represented by a factor graph model on G = (V, F, E ) with V = V, F = V, E = D + V, X = X. represent by a factor node each conditional probability moralization for conversion from BN to MRF from factor graph to Bayes network A factor model on G = (V, F, E) with alphabet X can be represented by a Bayes network G = (V, D ) with V = V and X = X. take a topological ordering, e.g. x 1,..., x n for each node i, starting from the first node, find a minimal set U {1,..., i 1} such that x i is conditionally independent of x {1,...,i 1}\U given x U. in general the resulting Bayes network is dense Graphical Models 2-18

19 Graphical Models 2-19 Bayes networks with observed variables V = H O Hidden variables: x = (x i ) i H Observed variables: y = (y i ) i O µ(x, y) = i H µ(x i x π(i) H, y π(i) O ) i O µ(y i x π(i) H, y π(i) O ) Typically interested in µ y (x) µ(x y) and arg max x µ y (x)

20 Graphical Models 2-20 Example Forensic Science [Kadane, Shum, A probabilistic analysis of the Sacco and Vanzetti evidence, 1996] [Taroni et al., Bayesian Networks and Probabilistic Inference in Forensic Science, 2006]

21 Graphical Models 2-21 Example diseases soft ORs symptoms Medical Diagnosis [M. Shwe, et al., Methods of Information in Medicine, 1991]

22 Graphical Models 2-22 Roadmap Cond. Indep. Factorization Graphical Graph Cond. Indep. µ(x) µ(x) Model G implied by G 1 x 1 {x 2, x 3 } x 4 ; Z ψa (x a ) FG Factor Markov 1 x 4 {} x 7 ; Z ψc (x C ) MRF Undirected Markov. ψi (x i x π(i) ) BN Directed Markov Any µ(x) can be represented by {FG,MRF,BN} A µ(x) can be represented by multiple {FG,MRF,BN} with multiple graphs (but same µ(x)) We want a simple graph representation (sparse, small alphabet size) Memory to store the graphical model Computations for inference µ(x) with some conditional independence structure can be represented by simple {FG,MRF,BN}

23 Graphical Models 2-23 Graphical models are incomparable: Each graphical model can represent independence constraints that others cannot We lose independence structure in conversion: Conversion results in a more complicated graph A cycle of conversion does not give the original graph back