Factor Graphs and Inference

Size: px

Start display at page:

Download "Factor Graphs and Inference"

Zoe Parsons
5 years ago
Views:

1 Factor Graph and Inerence Sargur Srihari 1

2 Topic 1. Factor Graph 1. Factor in probability ditribution. Deriving them rom graphical model. Eact Inerence Algorithm or Tree graph 1. The um-product algorithm 1. For inding marginal probabilitie. The ma-um algorithm 1. For etting o variable that maimize a probability. For value o that probability 3. Eact inerence in general graph

3 Factor in Joint Ditribution Joint ditribution written a Where i a ubet o the variable Special cae: p( Directed graph: are conditional ditribution K p p k pa k k1 p(a,b,cp(c a,bp(b ap(a Undirected graph: actor are potential unction over maimal clique p 1 ψ C C Z C P(A, B,C, D α ep[ ε 1 (A, B ε (B,C ε 3 (C, D ε 4 (D, A ] Partition unction Z i um o RHS over all value o A,B,C,D Log-linear Model 3

4 Factor Graph Motivation Joint ditribution Can be epreed a product o actor over ubet Factor graph make thi eplicit By introducing additional node or actor Sum-prod inerence algorithm (to be deined applicable to: undirected, directed tree, polytree Ha imple and general orm with actor graph Ued or determining marginal probabilitie 4

5 Factor Graph Eample Factorization p a 1, b 1, c, 3 d 3 Correponding actor graph Two actor a and b o the ame variable 5

6 Factor graph propertie They are bipartite ince 1. Two type o node. All link go between node o oppoite type Repreentable a two row o node Variable on top Factor node at bottom Other intuitive repreentation ued When derived rom directed/ undirected graph 6

7 Deriving actor graph rom Graphical Model Undirected Graph Directed Graph 7

8 Converion o Undirected to Factor Graph Step in converting ditribution epreed a undirected graph: 1. Create variable node correponding to node in original. Create actor node or maimal clique 3. Factor et equal to clique potential Several dierent actor graph poible rom ame ditribution Single clique potential Ψ 1,, 3 With actor 1,, 3 Ψ 1,, 3 With actor 8 a 1,, 3 b 1, Ψ 1,, 3

9 Converion o directed to actor graph Step 1. Variable node correpond to node in actor graph. Create actor node correponding to conditional ditribution Multiple actor graph poible rom ame graph Factorization p 1 p p 3 1, Single Factor 1,, 3 p 1 p p 3 1, With Factor a 1 p 1 b p 9 c 1,, 3 p 3 1,

10 Tree to Factor Graph Converion o directed or undirected tree to actor graph i a tree No loop Only one path between node In the cae o a directed polytree Converion to undirected graph ha loop due to moralization Converion again to actor graph reult in a tree Directed polytree Converted to Undirected Graph with loop Factor Graph 10

11 Removal o local cycle Local cycle in a directed graph having link connecting parent Can be removed on converion to actor graph By deining a actor unction Factor Graph with tree tructure 1,, 3 p 1 p 1 p 3 1, 11

12 Multiple actor graph or ame graph Factor graph are peciic about actorization A ully connected undirected graph Joint ditribution in two orm In general orm p 1,, 3 A a peciic actorization p a 1, b 1, 3 c, 3 1

13 The Sum-Product Algorithm Factor graph ramework ued to derive Powerul cla o eicient, eact inerence Algorithm or Tree- tructured Graph For evaluating local marginal over ubet o node Later modiied to Ma-um algorithm 13

14 Sum-Product Aumption Aume that all variable are dicrete Framework applicable to linear Gauian model where marginalization involve integration Belie Propagation i a pecial cae o um-product algorithm 14

15 Goal o Sum-Product Algorithm Firt convert tree tructured graph to a actor graph Can deal with directed/undirected uing ame ramework Goal i to eploit tructure o graph to: 1. Obtain eicient inerence algorithm to ind marginal. When everal marginal are required, to hare computation 15

16 Overview o um-product algorithm Finding p or a variable node Aume all variable are hidden Marginal i deined a Set o variable with removed Subtitute or p the actor-graph epreion Interchange ummation and product or eiciency 16 16

17 Epreing a ubtree Fragment o graph Tree tructure allow partitioning actor into group Set o node that are neighbor o Set o variable in ub-tree connected to via actor node Product o all actor in group aociated with actor 17

18 Meage rom actor node to variable node Subtituting into p p ne ne X F,X Where, a et o unction are deined a F,X X Which are viewed a meage rom actor node i product o all incoming meage at node p to variable node 18

$..,M Gmm, X m m ne( \ X 1 M m m m To evaluate meage ent by a actor node: Take product o incoming meage into actor node Multiply by actor aociated with the node Marginalize over all$

19 Meage rom variable to actor node Each actor Can itel be actored a F,X Reubtituting F,X, 1 i decribed by a actor ubgraph,..., M G M M, M..,1,...,M Gmm, X m m ne( \ X 1 M m m m To evaluate meage ent by a actor node: Take product o incoming meage into actor node Multiply by actor aociated with the node Marginalize over all variable aociated with incoming meage 19

$m \ Subtituting into epreion or m m m m l m m l ne \ Simply take product o incoming meage along other link m$

20 Evaluating meage rom variable node to actor node Making ue o ub-graph actorization G m m,x m F l m,x ml l ne m \ Subtituting into epreion or m m m m l m m l ne \ Simply take product o incoming meage along other link m 0

21 Sum-Product Algorithm Meage Goal i to calculate marginal or variable node Given by product o incoming meage along all link arriving at node I lea node i a variable node then it end 1 I lea node i actor node it end Variable Node Factor Node 1

22 Summary o um-product algorithm To evaluate the marginal p View node a the root o the actor graph Initiate meage at leave uing 1 The meage paing tep are applied recurively Until meage are propagated along every link And root node ha received meage rom all it neighbor Required marginal i evaluated uing p ne ne X F,X

23 Finding marginal or every node Need not run algorithm areh or each node More eicient approach Overlay multiple meage paing algorithm to get general meage paing algorithm Pick any variable or actor node a root Propagate meage rom root to the leave 3

24 Marginal or et o variable in a actor Marginal aociated with a actor i Product o meage arriving at actor node and Local actor at the node, i.e., p i ne( i i 4

25 Sum-product view a meage rom actor node Viewed purely a meage ent by actor node to other actor node Outgoing meage (blue arrow i obtained by Taking product o all incoming meage (green arrow Multiplying by actor Marginalized over variable and 1 5

26 Normalization I actor graph derived rom a directed graph then joint ditribution i correctly normalized So marginal will be normalized correctly I tated rom undirected graph Need normalization coeicient 1/Z Firt run um-product algorithm to ind unnormalized marginal ~ p i Normalize any one o thee marginal to obtain 1/Z 6

27 Sum product algorithm illutration Four-node actor graph with joint ditribution ~ p,,, a 1 b 3 c 4 Deignate 3 a root node 1 4 Then lea node are and Starting with lea node there are a equence o i meage 7

28 8 Meage rom lea node b c a b b c a b c c a a,,,

29 9 Meage rom root node to lea node c a b,,, c c b a c a b b b a b b

Max-Sum Inference Algorithm

Max-Sum Inference Algorithm Ma-Sum Inference Algorithm Sargur Srihari srihari@cedar.buffalo.edu 1 The ma-sum algorithm Sum-product algorithm Takes joint distribution epressed as a factor graph Efficiently finds marginals over component