Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGs Huizhen Yu jney.yu@s.helsinki.fi Dept. Computer Siene, Univ. of Helsinki Proilisti Moels, Spring, 2010 Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 1 / 22
Outline Direte Mrkov Properties n Mrkov Equivlent DAGs Other Direte Mrkov Properties n Equivlene DAG, Unirete Grphs, n Deomposle Grphs Mrkov Equivlene of DAGs Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 2 / 22
Other Direte Mrkov Properties n Equivlene Outline Direte Mrkov Properties n Mrkov Equivlent DAGs Other Direte Mrkov Properties n Equivlene DAG, Unirete Grphs, n Deomposle Grphs Mrkov Equivlene of DAGs Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 3 / 22
Other Direte Mrkov Properties n Equivlene For DAG G = (V, E), G m : the morl grph. Nottion G A : the sugrph inue y suset A of verties. p(α): the set of prents of α. h(α): the set of hilren of α. n(α): the set of nestors of α. e(α): the set of esennts of α. An nestrl set: set A suh tht p(v) A, v A. An(A): the miniml nestrl set ontining A; An(A) = A α A n(α). A well-orering: (α, β) E numer(α) < numer(β). pr(β): the set of preeessors of β, i.e., pr(β) = α numer(α) < numer(β). l(v): the Mrkov lnket of v, equl to ne(v), the set of neighors of v in G m ; i.e., l(v) = ne(v) = p(v) h(v) {w h(w) h(v) }. Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 4 / 22
Other Direte Mrkov Properties n Equivlene Min Results from Lst Leture For istriution P tht ftorizes reursively oring to DAG G, P ftorizes oring to the morl grph G m ; the mrginl istriution P A (X A ) ftorizes reursively oring to G An(A), from whih we otin the glol irete Mrkov property (DG): for ny isjoint susets A, B, S, A B S in `G An(A B S) m X A X B X S. We hve lso shown The -Seprtion onept; (DG) is equivlent to the -seprtion se Mrkov property. Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 5 / 22
Other Direte Mrkov Properties n Equivlene Lol Direte Mrkov Property Denote n(v) the set of non-esennts of v in G, i.e., n(v) = V \ (e(v) {v}). Consier v, its prents p(v), n its non-esennts n(v). Beuse An`{v} p(v) n(v) is the miniml nestrl set ontining these verties, h(v) An`{v} p(v) n(v) =. Eges in the morl grph `G An({v} p(v) n(v)) m re either from G An({v} p(v) n(v)) or from mrriges. The two fts ove imply tht v is not onnete iretly to n(v) \ p(v) in `G An({v} p(v) n(v)) m. So y (DG), X v X n(v) X p(v). (1) The property in (1) is lle the lol irete Mrkov property (DL). The isussion ove shows lso (DG) (DL). Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 6 / 22
Other Direte Mrkov Properties n Equivlene Illustrtion of (DL) Exmple Stu-Frm: L Ann Brin Ceily K p`dorothy = {Ann, Brin} e`dorothy = {Henry, John} Fre Dorothy Eri Gwenn Given the genotypes of Ann n Brin, Dorothy s genotype is inepenent of the genotype of ll the other noes in the grph elow. Henry John Irene L Ann Brin Ceily K Fre Dorothy Eri Gwenn Irene Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 7 / 22
Other Direte Mrkov Properties n Equivlene Equivlene etween Direte Mrkov Properties Rell tht for n unirete grph, (F) (G) (L) (P). By ontrst, for DAG, the Mrkov properties re ll equivlent: (DF) (DG) (DL) (DO), where these properties re s efine erlier: (DF) P ftorizes reursively oring to G, i.e., p(x) = Y p`x v x p(v). (DG) P oeys the glol irete Mrkov property with respet to G, i.e., A B S in `G An(A B S) m X A X B X S. (DL) P oeys the lol irete Mrkov property with respet to G, i.e., v V X v X n(v) X p(v). (DO) P oeys the orere irete Mrkov property with respet to G, i.e., with respet to ny well-orering, X v X pr(v) X p(v). We verifie (DF) (DG) (DL) n (DO) (DF). It is lso strightforwr to show (DL) (DO) (exerise). Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 8 / 22
Other Direte Mrkov Properties n Equivlene Exmples from Erlier Letures Revisite In Le. 2, we verifie the following sttements using equtions. We n now verify them using only grphs. If (X 1,..., X n) is Mrkov hin, then (X n1, X n2,..., X nk ) is lso Mrkov hin, where 1 < n 1 <... < n k n: (X n,..., X 1) is lso Mrkov hin: 1 2 n 1 n 2 n k-1 n k 1 2 n-1 n 1 2 n 1 n 2 n k-1 n k 1 2 n-1 n X j X j (X j 1, X j+1 ), where X j enotes the olletion of ll the vriles ut X j : 1 2 j-1 j j+1 n 1 2 j-1 j j+1 n In n HMM, the oservtion vriles Y 1,..., Y n re generlly fully epenent: Y1 Y2 Yn-1 Yn Y1 Y2 Y n-1 Yn Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 9 / 22
DAG, Unirete Grphs, n Deomposle Grphs Outline Direte Mrkov Properties n Mrkov Equivlent DAGs Other Direte Mrkov Properties n Equivlene DAG, Unirete Grphs, n Deomposle Grphs Mrkov Equivlene of DAGs Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 10 / 22
DAG, Unirete Grphs, n Deomposle Grphs Some Oservtions Rell tht if P ftorizes oring to G m, P oes not neessrily ftorize reursively oring to G: p(x) = Y φ C (x C ) p(x) = Y p`x v x p(v). C C(G m ) v V Exmple: G α β α G m β P(X α, X β, X γ) with p(x) = exp{x αx γ + x β x γ} ftorizes oring to G m, ut γ γ p(x) p(x α)p(x β )p(x γ x α, x β ) euse X α X β. Conversely, unirete grphs n lso represent onitionl inepenene reltions tht nnot e represente y DAGs: e.g., {} {} {, } {} {} {, } Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 11 / 22
DAG, Unirete Grphs, n Deomposle Grphs Some Oservtions But for hins n more generlly, roote trees, the irete n unirete version seems to e equivlent: ftoriztion w.r.t. one implies ftoriztion w.r.t. the other. 1 2 n-1 n 1 2 n-1 n Y1 Y2 Yn-1 Yn Y1 Y2 Y n-1 Yn e e f f Are there other types of grphs for whih this is true? Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 12 / 22
DAG, Unirete Grphs, n Deomposle Grphs Deomposle Grphs: Definition A triple (A, B, C) of isjoint susets of the vertex set V of n unirete grph G is si to eompose G, if V = A B C, n the following two onitions hol: 1. C seprtes A from B; 2. C is omplete suset of V. If oth A n B re non-empty, we sy tht the eomposition is proper. We sy tht n unirete grph G is eomposle if either: (i) it is omplete, or (ii) it possesses proper eomposition (A, B, C) suh tht oth sugrphs G A C n G B C re eomposle. Note tht this efinition is reursive. Exmples: hins, trees, n e f Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 13 / 22
DAG, Unirete Grphs, n Deomposle Grphs Deomposle Grphs: Ftoriztion of Distriutions Proposition 1: Assume tht (A, B, S) eomposes n unirete grph G = (V, E). Then istriution P ftorizes oring to G if n only if oth P A S n P B S ftorize oring to G A S n G B S respetively n P stisfies p(x) = p A S(x A S ) p B S (x B S ). p S (x S ) Implition: If the grph is eomposle, then the mrginl istriutions P A S n P B S mit further ftoriztion in the ove form, n P ftorizes reursively in terms of these mrginls. Exmple: For Mrkov hin X = (X 1,..., X n), P(X ) n e expresse s p(x) = p(x1, x2)p(x2, x3) p(xn 1, xn). p(x 2)p(x 3) p(x n 1) Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 14 / 22
DAG, Unirete Grphs, n Deomposle Grphs Proof of Proposition 1 Let C(G) enote the set of liques of G. Suppose P ftorizes oring to G, i.e.,, p(x) = Y φ C (x C ), for some funtions φ C. C C(G) Sine (A, B, S) eomposes G, ll the liques tht re not susets of A S must e susets of B S, so tht p(x) = Y C C(G A S ) φ C (x C ) Y C C(G B S ) φ C (x C ) = h(x A S ) k(x B S ), for some funtions h, k. It follows then p(x A S ) = X h(x A S ) k(x B S ) = h(x A S ) k(x S ), x B where k(x S ) = X k(x B S ); x B p(x B S ) = X x A h(x A S ) k(x B S ) = h(x S ) k(x B S ), where h(x S ) = X x A h(x A S ); So p(x S ) = X x A X p A S (x A S ) p B S (x B S ) p S (x S ) The onverse is evient. x B h(x A S ) k(x B S ) = h(x S ) k(x S ). = h(x A S ) k(x S ) h(x S ) k(x B S ) h(x S ) k(x S ) = h(x A S ) k(x B S ) = p(x). Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 15 / 22
DAG, Unirete Grphs, n Deomposle Grphs Deomposle Grphs: Direte or Unirete? Definition: A DAG G = (V, E) is si to e perfet if for ll v V, p(v) is omplete suset of G, in other wors, if its unirete version G = G m. Note: G of perfet DAG is eomposle. (How to see this?) Proposition 2: Let G e perfet DAG n G its unirete version. Then istriution P is irete Mrkov with respet to G if n only if it ftorizes oring to G. Implition: Proilisti Inepenenes/epenenes Deomposle Moels DAG Moels Unirete Moels Chin Grph Moels Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 16 / 22
DAG, Unirete Grphs, n Deomposle Grphs Proof of Proposition 2 Proof: Sine G = G m, if P ftorizes reursively oring to G, then it ftorizes oring to G. Conversely, if P ftorizes oring to G, we show P reursively ftorizes oring to G y inution on the numer of verties V. For V = 1, the onlusion hols trivilly. For V > 1, onsier terminl vertex α. Then, (V \ ({α} p(α)), {α}, p(α)) eomposes D. So y Proposition 1, P n e expresse s p(x) = p(x V \{α}) p(x α, x p(α) ) p(x p(α) ) = p(x V \{α} )p(x α x p(α) ), (2) n P V \{α} ftorizes oring to GV \{α}. By inution, P V \{α} ftorizes reursively oring to G V \{α} : p(x V \{α} ) = Y p(x v x p(v) ). v V \{α} Together with Eq. (2) this shows tht P ftorizes reursively oring to G. Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 17 / 22
Mrkov Equivlene of DAGs Outline Direte Mrkov Properties n Mrkov Equivlent DAGs Other Direte Mrkov Properties n Equivlene DAG, Unirete Grphs, n Deomposle Grphs Mrkov Equivlene of DAGs Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 18 / 22
Mrkov Equivlene of DAGs Mrkov Equivlene of DAGs Terminologies for DAG G: Skeleton of G: G, the unirete version of G. A omplex (lso lle n immorlity): sugrph of G inue y three verties (α, γ, β), of the form shown on the right. I.e., the two prents α, β re not joine in G. α γ β Definition: We sy tht two DAG G n e G re Mrkov equivlent if they represent the sme onitionl inepenene reltions. Theorem: Two DAG G n e G re Mrkov equivlent if n only if they hve the sme skeleton n the sme omplexes. This theorem hs importnt implition for utomti moel seletion: Sine (DG) (DF), the set of istriutions ssoite with G is the sme s tht ssoite with ny DAG Mrkov equivlent to G. When eges o not hve usl interprettions, it woul e in vin to try to istinguish etween equivlent moels, n we nee to ompre equivlent lsses of DAGs inste. (The representtive grph of n equivlent lss is hin grph, interestingly.) Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 19 / 22
Mrkov Equivlene of DAGs Exmples of Mrkov Equivlent DAGs An equivlent lss onsisting of three DAGs n their representtive (from Anersson et l. 1997): Wht hppene to the other 5 possiilities? Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 20 / 22
Mrkov Equivlene of DAGs Proof of the Mrkov Equivlene Theorem for DAG Proof of suffiieny: if G n G e hve the sme skeleton n the sme omplexes, then they hve the sme morl grphs for ll sugrphs inue y the sme set of verties. This mens tht for ny isjoint susets A, B, S of verties, A B S in `G An(A B S) m A B S in `e GAn(A B S) m. So G n e G represent the sme onitionl inepenene reltions. Proof of neessity: we rgue this y using ounterexmples. Sme skeleton: suppose there is n ege etween α n β in e G ut not in G. Then α, β is non-jent in `G An({α,β}) m, so α β An({α, β}) \ {α, β}. Assume x { 1, 1} V. Let p(x) exp{x αx β }. Then P ftorizes oring to e G, ut X α n X β re not inepenent given X An({α,β})\{α,β}, ontrition. Sme omplexes: suppose there is omplex inue y (α, γ, β) in G, ut in eg, the onnetion etween them is ifferent. Agin, α, β is non-jent in `GAn({α,β}) m, so α β An({α, β}) \ {α, β}. Assume x { 1, 1} V. Let p(x) exp{x αx γ + x β x γ}. Then P ftorizes oring to e G, ut X α n X β re not inepenent without X γ given, ontrition. Note: This theorem is shown y Verm n Perl (1991) n inepenently y Fryenerg (1990) (for hin grphs). In the ove we followe the ltter proof, whih uses (DG) inste of -seprtion. Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 21 / 22
Mrkov Equivlene of DAGs Further Reings For irete Mrkov properties n eomposle grphs: 1. Roert G. Cowell et l. Proilisti Networks n Expert Systems, Springer, 2007. Chp. 5.2, 5.3. Huizhen Yu (U.H.) Byesin Networks: Direte Mrkov Properties (Cont ) n Mrkov Equivlent DAGsFe. 11 22 / 22