Graphical Models. Pradeep Ravikumar Department of Computer Science The University of Texas at Austin
|
|
- Agatha Lang
- 5 years ago
- Views:
Transcription
1 Graphical Models Pradeep Ravikumar Department of Computer Science The University of Texas at Austin
2 Useful References Graphical models, exponential families, and variational inference. M. J. Wainwright and M. I. Jordan. Foundations and Trends in Machine Learning, Vol. 1, Numbers 1--2, pp , December (Available for free online) Graphical Models with R. Soren Hojsgaard, David Edwards, Steffen Lauritzen Probabilistic Graphical Models: Principles and Techniques. D. Koller and N. Friedman.
3 Applications of Graphical Models Computer vision and graphics Natural language processing Information retrieval Robotic control Computational biology Medical diagnosis Finance and economics Error-control codes Decision making under uncertainty...
4 Speech Recognition
5
6 Bioinformatics
7
8
9
10 Computer Vision
11
12
13 Abstraction Layers and Interfaces Internet Protocol Layers Computer Architecture Layers
14 Abstraction Layers and Interfaces Internet Protocol Layers Computer Architecture Layers Interface layer for machine learning?
15 Abstraction Layers and Interfaces Internet Protocol Layers Computer Architecture Layers Interface layer for machine learning? Graphical Models!
16 Graphical Models: The Why Compact way of representing distributions among random variables
17 Graphical Models: The Why Compact way of representing distributions among random variables Visually appealing way (accessible to domain experts) of representing distributions among random variables
18 Graphical Models: The Why Compact way of representing distributions among random variables Visually appealing way (accessible to domain experts) of representing distributions among random variables They represent distributions among random variables probability theory
19 Graphical Models: The Why Compact way of representing distributions among random variables Visually appealing way (accessible to domain experts) of representing distributions among random variables They represent distributions among random variables probability theory They use graphs to do so graph theory
20 Graphical Models: The Why Compact way of representing distributions among random variables Visually appealing way (accessible to domain experts) of representing distributions among random variables They represent distributions among random variables probability theory They use graphs to do so graph theory A marriage of graph and probability theory
21 Graphical Models: Representation
22 Joint Distributions Consider a set of random variables {X 1,X 2,...,X n } Let x i be a realization of random variable X i ; x i may be real, discrete, vector in vector space; for now assume variables are discrete. Probability Mass Function p(x 1,...,x n ):=P (X 1 = x 1,...,X n = x n ). Shorthand: X =(X 1,...,X n ), x =(x 1,...,x n ), so that p(x) =P (X = x).
23
24
25
26
27 Directed Graphs Let V = {1, 2,...,n}. Given a set of non-negative functions {f i (x i,x i ): i 2 V }, each of which sum to one as a function of first argument, we define a joint probability distribution: p(x 1,...,x n ):= Exercise: Calculate p(x i x i ). ny f i (x i,x i ). i=1
28 Directed Graphs Let V = {1, 2,...,n}. Given a set of non-negative functions {f i (x i,x i ): i 2 V }, each of which sum to one as a function of first argument, we define a joint probability distribution: p(x 1,...,x n ):= Exercise: Calculate p(x i x i ). ny f i (x i,x i ). i=1 Can be shown that p(x i x i )=f i (x i,x i ).
29 Directed Graphs Let V = {1, 2,...,n}. Given a set of non-negative functions {f i (x i,x i ): i 2 V }, each of which sum to one as a function of first argument, we define a joint probability distribution: p(x 1,...,x n ):= Exercise: Calculate p(x i x i ). ny f i (x i,x i ). i=1 Can be shown that p(x i x i )=f i (x i,x i ).
30
31
32
33 Representational Economy Directed graphical model distributions thus can be represented or stored much more cheaply than a general distribution What s the catch? The graphical model distribution satisfy certain conditional independence assumptions
34
35
36
37
38
39
40
41
42
43 Graphical Models and Conditional Independences A Graphical Model is a family of probability distributions. Given a graph, specify any set of conditional distributions of nodes given parents.
44 Graphical Models and Conditional Independences A Graphical Model is a family of probability distributions. Given a graph, specify any set of conditional distributions of nodes given parents. Any such distribution satisfies conditional independences
45 Graphical Models and Conditional Independences A Graphical Model is a family of probability distributions. Given a graph, specify any set of conditional distributions of nodes given parents. Any such distribution satisfies conditional independences We will now provide an algorithm (Bayes-Ball, d-separation) to list all conditional independence statements satisfied by the given family of graphical model distributions
46 Three Canonical Graphs
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66 Is there a graph-theoretic way to write down all conditional independence statements given graph?
67
68 Blocking Rules I X Y Z X Y Z
69
70
71
72
73
74
75
76
77
78
79
80 Undirected Graphical Models
81
82
83 Undirected Graphical Models In the directed graphical model case, we started with a factorized form before discussing conditional independence assumptions because the former was more natural (conditional probabilities vs Bayes-Ball algorithm) In the undirected graphical model case, the conditional independence assertions are much more natural and intuitive Just depend on classical graph separation (contrast with blocking rules) X_A is cond. independent of X_C given X_B, if nodes in X_A are separated from nodes in X_C given nodes in X_B Remove nodes in X_B: no path from nodes in X_A to those in X_B
84
85 Undirected Graphical Models: Parameterization
86 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above
87 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above the set of distributions that satisfy all these assertions is the undirected graphical model family given the graph
88 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above the set of distributions that satisfy all these assertions is the undirected graphical model family given the graph We would like to obtain a local parameterization of an undirected graphical model
89 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above the set of distributions that satisfy all these assertions is the undirected graphical model family given the graph We would like to obtain a local parameterization of an undirected graphical model Why? For compact representation of joint distribution!
90 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above the set of distributions that satisfy all these assertions is the undirected graphical model family given the graph We would like to obtain a local parameterization of an undirected graphical model Why? For compact representation of joint distribution! For directed graphs, the parameterization was based on local conditional probabilities of nodes given their parents
91 Undirected Graphical Models: Parameterization Given an undirected graph, we can define the set of conditional independence assertions using graph separation as above the set of distributions that satisfy all these assertions is the undirected graphical model family given the graph We would like to obtain a local parameterization of an undirected graphical model Why? For compact representation of joint distribution! For directed graphs, the parameterization was based on local conditional probabilities of nodes given their parents local had the interpretation of a set consisting of a node and its parents
92 Undirected Graphical Models: Parameterization We would like to obtain a local parameterization of an undirected graphical model Conditional probabilities of nodes given neighbors? Hard to ensure consistency of such conditional probabilities across nodes, so won t be able to choose such functions independently for each node
93 Undirected Graphical Models: Parameterization We would like to obtain a local parameterization of an undirected graphical model Conditional probabilities of nodes given neighbors? Hard to ensure consistency of such conditional probabilities across nodes, so won t be able to choose such functions independently for each node Their product need not be a valid joint distribution
94 Undirected Graphical Models: Parameterization How do we use conditional probabilities in the undirected graphical model case (where there are no parents)? Conditional probabilities of nodes given neighbors? Hard to ensure consistency of such conditional probabilities across nodes, so won t be able to choose such functions independently for each node Their product need not be a valid joint distribution Better to not use conditional probabilities, but to use product of some other local functions that can be chosen independently
95 Conditional Independence and Factorization If X A?? X C X B,then P (X A X B,X C )=P (X A X C ) P (X A,X B,X C )=P (X A X C ) P (X B,X C ) The joint distribution P (X A,X B,X C )=f(x A,X B ) g(x B,X C ), for some functions f and g. Conditional Independence entails factorization!
96 Conditional Independence and Factorization If X A?? X C X B,then P (X A X B,X C )=P (X A X C ) P (X A,X B,X C )=P (X A X C ) P (X B,X C ) The joint distribution P (X A,X B,X C )=f(x A,X B ) g(x B,X C ), for some functions f and g. Conditional Independence entails factorization! If there is no edge between X i and X j, then the joint distribution cannot have a factor such as (X i,x j,x k ). Factors of the joint distribution can depend only on variables within a fully connected subgraph, also called a clique. Without loss of generality, assume factors only over maximum cliques
97 Conditional Independence and Factorization If X A?? X C X B,then P (X A X B,X C )=P (X A X C ) P (X A,X B,X C )=P (X A X C ) P (X B,X C ) The joint distribution P (X A,X B,X C )=f(x A,X B ) g(x B,X C ), for some functions f and g. Conditional Independence entails factorization! If there is no edge between X i and X j, then the joint distribution cannot have a factor such as (X i,x j,x k ). Factors of the joint distribution can depend only on variables within a fully connected subgraph, also called a clique. Without loss of generality, assume factors only over maximum cliques
98 Conditional Independence and Factorization If X A?? X C X B,then P (X A X B,X C )=P (X A X C ) P (X A,X B,X C )=P (X A X C ) P (X B,X C ) The joint distribution P (X A,X B,X C )=f(x A,X B ) g(x B,X C ), for some functions f and g. Conditional Independence entails factorization! If there is no edge between X i and X j, then the joint distribution cannot have a factor such as (X i,x j,x k ). Factors of the joint distribution can depend only on variables within a fully connected subgraph, also called a clique. Without loss of generality, assume factors only over maximum cliques
99
100 Undirected Graphical Models: Factorization x x x x X 2 X 4 X 1 X 6 0 x 1 1 x X 3 0 x 3 1 x X 5 0 x 5 1 x x 2
101 What do potential functions mean? We just discussed how local conditional probabilities are not satisfactory as potential functions Why not replace potential functions C (x C ) with marginal probabilities p(x C )?
102
103 What do potential functions mean? Potential functions are neither conditional nor marginal probabilities, and do not have a local probabilistic interpretation But do often have a pre-probabilistic interpretation as agreement, constraint or energy Potential function favors certain configurations by assigning larger value
104
105 Dual Characterization of Undirected Graphical Models Characterization I: Given a graph G, all distributions expressed as product of potential functions over maximal cliques Characterization II: Given a graph G, all distributions that satisfy all conditional independence statements specified by graph separation Hammersley Clifford Theorem: Characterizations are equivalent! As in the directed case, a profound link between probability theory and graph theory
106 Beyond Directed/Undirected Graphical Models
107
108
109
110
111
112
113
114 Factorization Factorization is a richer concept than conditional independence There exist families of distributions defined in terms of factorizations that cannot be captured solely within (un)directed graphical model formalism. From point of view of graphical model inference algorithms, not a problem: would work for any distribution within any sub-family defined on graph. But by missing out on algebraic structure, might not have the most e - cient inference algorithm possible
115 Factorization Factorization is a richer concept than conditional independence There exist families of distributions defined in terms of factorizations that cannot be captured solely within (un)directed graphical model formalism. From point of view of graphical model inference algorithms, not a problem: would work for any distribution within any sub-family defined on graph. But by missing out on algebraic structure, might not have the most e - cient inference algorithm possible We will soon see factor graphs, a graphical representation that makes such reduced parameterizations explicit, and in particular could capture the factorized family example earlier Factor graphs provide nothing new in terms of exploiting conditional independences, but provide a way to exploit algebraic relationships.
116
117 Factor Graphs Given a set of variables {x 1,...,x n }, let C denote a set of subsets of {1,...,n}. Example: C = {{1, 3}, {3, 4}, {2, 4, 5}, {1, 3}}. Denote: C = {C s : s 2 F}. To each index s 2 F, associate factor f s (x Cs ), function on variables indexed by C s. Example above: with index set F = {a, b, c, d}, factors are f a (x 1,x 3 ),f b (x 3,x 4 ),f 3 (x 2,x 4,x 5 ),f d (x 1,x 3 ) Note that C is an arbitrary collection of subsets of indices; need not correspond to cliques of some graph (note that we have not even specified any graph at this point!)
118 Factor Graphs Such factorized forms occur in many areas of mathematics, and has applications outside probability theory Our interest is in application to probability distributions of course Can see that both undirected and directed graphical models can also be written as a product of factors We have yet not defined the factor graph a graphical representation of the factored distribution above
119 Factor Graph Bipartite graph G(V, F, E), where vertices V index variables, the vertices F index the factors. The edges E: each factor node s 2 F linked to all variable nodes in C s ; these are the only edges.
120
121
122 Graphical Models: Inference
123 Probabilistic Inference Let E and F be disjoint subsets of nodes of a graphical model; with XE and XF the disjoint subsets of variables corresponding to E, F We want to calculate P(xF xe) for arbitrary subsets E and F This is the general probabilistic inference problem We will focus initially on a single query node F, and directed graphical models
124
125
126 Simplest Case X Y
127 Simplest Case X X Y Y P(Y X) is specified directly by the conditional probability table
128
129
130
131
132
133
134 Inference Algorithms Exact Algorithms: these solve the inference problem exactly Caveat: Time taken is in general exponential in tree-width of graph (which is typically not constant, and scales poorly with the number of nodes) Variable Elimination, Sum-Product (for tree-structured graphs) Approximate algorithms: these solve the inference problem approximately While time taken is linear/quadratic in number of nodes, even to obtain methods with approximation guarantees can be shown to be NP-hard Sum-Product (for general graphs)
135 Sum Product Message-Passing Protocol: A node can send a message to a neighboring node when (and only when) it has received messages from all of its remaining neighbors
136 Sum Product Message-Passing Protocol: A node can send a message to a neighboring node when (and only when) it has received messages from all of its remaining neighbors Two ways to implement this protocol: Parallel From leaves to root, and from root to leaves (as in previous diagram, this will provide messages along both directions for any edge)
137
138 Sum Product: Parallel (a)
139 Sum Product: Parallel (a) (b)
140
141 Sum Product: Parallel (a) (b) (c) (d)
142
143 Graphical Models: Learning
144 We will focus on pairwise graphical models p(x;,g)= 1 Z( ) exp (s,t) E(G) st st (X s,x t ) st(x s,x t ) : arbitrary potential functions Ising x s x t Potts I(x s = x t ) Indicator I(x s,x t = j, k)
145
146
147
148
149
150 Learning Graphical Models
151 Learning Graphical Models Two Step Procedures:
152 Learning Graphical Models Two Step Procedures: 1. Model Selection; estimate graph structure
153 Learning Graphical Models Two Step Procedures: 1. Model Selection; estimate graph structure 2. Parameter Inference given graph structure
154 Learning Graphical Models Two Step Procedures: 1. Model Selection; estimate graph structure 2. Parameter Inference given graph structure Score Based Approaches: search over space of graphs, with a score for graph based on parameter inference
155 Learning Graphical Models Two Step Procedures: 1. Model Selection; estimate graph structure 2. Parameter Inference given graph structure Score Based Approaches: search over space of graphs, with a score for graph based on parameter inference Constraint-based Approaches: estimate individual edges by hypothesis tests for conditional independences
156 Learning Graphical Models Two Step Procedures: 1. Model Selection; estimate graph structure 2. Parameter Inference given graph structure Score Based Approaches: search over space of graphs, with a score for graph based on parameter inference Constraint-based Approaches: estimate individual edges by hypothesis tests for conditional independences Caveats: (a) difficult to provide guarantees for estimators; (b) estimators are NP- Hard
157 Learning Graphical Models State of the art methods are based on estimating neighborhoods Via high-dimensional statistical model estimation Via high-dimensional hypothesis tests
Lecture 3: Conditional Independence - Undirected
CS598: Graphical Models, Fall 2016 Lecture 3: Conditional Independence - Undirected Lecturer: Sanmi Koyejo Scribe: Nate Bowman and Erin Carrier, Aug. 30, 2016 1 Review for the Bayes-Ball Algorithm Recall
More informationGraphical Models. David M. Blei Columbia University. September 17, 2014
Graphical Models David M. Blei Columbia University September 17, 2014 These lecture notes follow the ideas in Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan. In addition,
More informationThe Basics of Graphical Models
The Basics of Graphical Models David M. Blei Columbia University September 30, 2016 1 Introduction (These notes follow Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan.
More informationCS242: Probabilistic Graphical Models Lecture 3: Factor Graphs & Variable Elimination
CS242: Probabilistic Graphical Models Lecture 3: Factor Graphs & Variable Elimination Instructor: Erik Sudderth Brown University Computer Science September 11, 2014 Some figures and materials courtesy
More informationLoopy Belief Propagation
Loopy Belief Propagation Research Exam Kristin Branson September 29, 2003 Loopy Belief Propagation p.1/73 Problem Formalization Reasoning about any real-world problem requires assumptions about the structure
More informationComputer Vision Group Prof. Daniel Cremers. 4. Probabilistic Graphical Models Directed Models
Prof. Daniel Cremers 4. Probabilistic Graphical Models Directed Models The Bayes Filter (Rep.) (Bayes) (Markov) (Tot. prob.) (Markov) (Markov) 2 Graphical Representation (Rep.) We can describe the overall
More information2. Graphical Models. Undirected graphical models. Factor graphs. Bayesian networks. Conversion between graphical models. Graphical Models 2-1
Graphical Models 2-1 2. Graphical Models Undirected graphical models Factor graphs Bayesian networks Conversion between graphical models Graphical Models 2-2 Graphical models There are three families of
More informationLecture 5: Exact inference. Queries. Complexity of inference. Queries (continued) Bayesian networks can answer questions about the underlying
given that Maximum a posteriori (MAP query: given evidence 2 which has the highest probability: instantiation of all other variables in the network,, Most probable evidence (MPE: given evidence, find an
More informationD-Separation. b) the arrows meet head-to-head at the node, and neither the node, nor any of its descendants, are in the set C.
D-Separation Say: A, B, and C are non-intersecting subsets of nodes in a directed graph. A path from A to B is blocked by C if it contains a node such that either a) the arrows on the path meet either
More informationLecture 5: Exact inference
Lecture 5: Exact inference Queries Inference in chains Variable elimination Without evidence With evidence Complexity of variable elimination which has the highest probability: instantiation of all other
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Eric Xing Lecture 14, February 29, 2016 Reading: W & J Book Chapters Eric Xing @
More informationPart II. C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
Part II C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Converting Directed to Undirected Graphs (1) Converting Directed to Undirected Graphs (2) Add extra links between
More informationComputer Vision Group Prof. Daniel Cremers. 4. Probabilistic Graphical Models Directed Models
Prof. Daniel Cremers 4. Probabilistic Graphical Models Directed Models The Bayes Filter (Rep.) (Bayes) (Markov) (Tot. prob.) (Markov) (Markov) 2 Graphical Representation (Rep.) We can describe the overall
More informationGraphical models and message-passing algorithms: Some introductory lectures
Graphical models and message-passing algorithms: Some introductory lectures Martin J. Wainwright 1 Introduction Graphical models provide a framework for describing statistical dependencies in (possibly
More informationFMA901F: Machine Learning Lecture 6: Graphical Models. Cristian Sminchisescu
FMA901F: Machine Learning Lecture 6: Graphical Models Cristian Sminchisescu Graphical Models Provide a simple way to visualize the structure of a probabilistic model and can be used to design and motivate
More informationExpectation Propagation
Expectation Propagation Erik Sudderth 6.975 Week 11 Presentation November 20, 2002 Introduction Goal: Efficiently approximate intractable distributions Features of Expectation Propagation (EP): Deterministic,
More information3 : Representation of Undirected GMs
0-708: Probabilistic Graphical Models 0-708, Spring 202 3 : Representation of Undirected GMs Lecturer: Eric P. Xing Scribes: Nicole Rafidi, Kirstin Early Last Time In the last lecture, we discussed directed
More informationUndirected Graphical Models. Raul Queiroz Feitosa
Undirected Graphical Models Raul Queiroz Feitosa Pros and Cons Advantages of UGMs over DGMs UGMs are more natural for some domains (e.g. context-dependent entities) Discriminative UGMs (CRF) are better
More informationBipartite Roots of Graphs
Bipartite Roots of Graphs Lap Chi Lau Department of Computer Science University of Toronto Graph H is a root of graph G if there exists a positive integer k such that x and y are adjacent in G if and only
More informationRecall from last time. Lecture 4: Wrap-up of Bayes net representation. Markov networks. Markov blanket. Isolating a node
Recall from last time Lecture 4: Wrap-up of Bayes net representation. Markov networks Markov blanket, moral graph Independence maps and perfect maps Undirected graphical models (Markov networks) A Bayes
More informationECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning
ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning Topics Bayes Nets: Inference (Finish) Variable Elimination Graph-view of VE: Fill-edges, induced width
More information6 : Factor Graphs, Message Passing and Junction Trees
10-708: Probabilistic Graphical Models 10-708, Spring 2018 6 : Factor Graphs, Message Passing and Junction Trees Lecturer: Kayhan Batmanghelich Scribes: Sarthak Garg 1 Factor Graphs Factor Graphs are graphical
More informationChapter 8 of Bishop's Book: Graphical Models
Chapter 8 of Bishop's Book: Graphical Models Review of Probability Probability density over possible values of x Used to find probability of x falling in some range For continuous variables, the probability
More informationCS242: Probabilistic Graphical Models Lecture 2B: Loopy Belief Propagation & Junction Trees
CS242: Probabilistic Graphical Models Lecture 2B: Loopy Belief Propagation & Junction Trees Professor Erik Sudderth Brown University Computer Science September 22, 2016 Some figures and materials courtesy
More informationPATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Bayesian Curve Fitting (1) Polynomial Bayesian
More informationOSU CS 536 Probabilistic Graphical Models. Loopy Belief Propagation and Clique Trees / Join Trees
OSU CS 536 Probabilistic Graphical Models Loopy Belief Propagation and Clique Trees / Join Trees Slides from Kevin Murphy s Graphical Model Tutorial (with minor changes) Reading: Koller and Friedman Ch
More informationThese notes present some properties of chordal graphs, a set of undirected graphs that are important for undirected graphical models.
Undirected Graphical Models: Chordal Graphs, Decomposable Graphs, Junction Trees, and Factorizations Peter Bartlett. October 2003. These notes present some properties of chordal graphs, a set of undirected
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Suggested Reading: Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Probabilistic Modelling and Reasoning: The Junction
More informationCOS 513: Foundations of Probabilistic Modeling. Lecture 5
COS 513: Foundations of Probabilistic Modeling Young-suk Lee 1 Administrative Midterm report is due Oct. 29 th. Recitation is at 4:26pm in Friend 108. Lecture 5 R is a computer language for statistical
More informationLecture 4: Undirected Graphical Models
Lecture 4: Undirected Graphical Models Department of Biostatistics University of Michigan zhenkewu@umich.edu http://zhenkewu.com/teaching/graphical_model 15 September, 2016 Zhenke Wu BIOSTAT830 Graphical
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 5 Inference
More informationProbabilistic Graphical Models
Overview of Part One Probabilistic Graphical Models Part One: Graphs and Markov Properties Christopher M. Bishop Graphs and probabilities Directed graphs Markov properties Undirected graphs Examples Microsoft
More informationExact Inference: Elimination and Sum Product (and hidden Markov models)
Exact Inference: Elimination and Sum Product (and hidden Markov models) David M. Blei Columbia University October 13, 2015 The first sections of these lecture notes follow the ideas in Chapters 3 and 4
More informationMachine Learning. Sourangshu Bhattacharya
Machine Learning Sourangshu Bhattacharya Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Curve Fitting Re-visited Maximum Likelihood Determine by minimizing sum-of-squares
More informationarxiv: v1 [cs.dm] 21 Dec 2015
The Maximum Cardinality Cut Problem is Polynomial in Proper Interval Graphs Arman Boyacı 1, Tinaz Ekim 1, and Mordechai Shalom 1 Department of Industrial Engineering, Boğaziçi University, Istanbul, Turkey
More information4 Factor graphs and Comparing Graphical Model Types
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms for Inference Fall 2014 4 Factor graphs and Comparing Graphical Model Types We now introduce
More informationV,T C3: S,L,B T C4: A,L,T A,L C5: A,L,B A,B C6: C2: X,A A
Inference II Daphne Koller Stanford University CS228 Handout #13 In the previous chapter, we showed how efficient inference can be done in a BN using an algorithm called Variable Elimination, that sums
More informationComputer Vision Group Prof. Daniel Cremers. 4a. Inference in Graphical Models
Group Prof. Daniel Cremers 4a. Inference in Graphical Models Inference on a Chain (Rep.) The first values of µ α and µ β are: The partition function can be computed at any node: Overall, we have O(NK 2
More informationComputer vision: models, learning and inference. Chapter 10 Graphical Models
Computer vision: models, learning and inference Chapter 10 Graphical Models Independence Two variables x 1 and x 2 are independent if their joint probability distribution factorizes as Pr(x 1, x 2 )=Pr(x
More informationCh9: Exact Inference: Variable Elimination. Shimi Salant, Barak Sternberg
Ch9: Exact Inference: Variable Elimination Shimi Salant Barak Sternberg Part 1 Reminder introduction (1/3) We saw two ways to represent (finite discrete) distributions via graphical data structures: Bayesian
More informationIEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 51, NO. 7, JULY A New Class of Upper Bounds on the Log Partition Function
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 51, NO. 7, JULY 2005 2313 A New Class of Upper Bounds on the Log Partition Function Martin J. Wainwright, Member, IEEE, Tommi S. Jaakkola, and Alan S. Willsky,
More information1 : Introduction to GM and Directed GMs: Bayesian Networks. 3 Multivariate Distributions and Graphical Models
10-708: Probabilistic Graphical Models, Spring 2015 1 : Introduction to GM and Directed GMs: Bayesian Networks Lecturer: Eric P. Xing Scribes: Wenbo Liu, Venkata Krishna Pillutla 1 Overview This lecture
More informationStatistical and Learning Techniques in Computer Vision Lecture 1: Markov Random Fields Jens Rittscher and Chuck Stewart
Statistical and Learning Techniques in Computer Vision Lecture 1: Markov Random Fields Jens Rittscher and Chuck Stewart 1 Motivation Up to now we have considered distributions of a single random variable
More informationWorkshop report 1. Daniels report is on website 2. Don t expect to write it based on listening to one project (we had 6 only 2 was sufficient
Workshop report 1. Daniels report is on website 2. Don t expect to write it based on listening to one project (we had 6 only 2 was sufficient quality) 3. I suggest writing it on one presentation. 4. Include
More informationDiscrete Wiskunde II. Lecture 6: Planar Graphs
, 2009 Lecture 6: Planar Graphs University of Twente m.uetz@utwente.nl wwwhome.math.utwente.nl/~uetzm/dw/ Planar Graphs Given an undirected graph (or multigraph) G = (V, E). A planar embedding of G is
More informationFinding Non-overlapping Clusters for Generalized Inference Over Graphical Models
1 Finding Non-overlapping Clusters for Generalized Inference Over Graphical Models Divyanshu Vats and José M. F. Moura arxiv:1107.4067v2 [stat.ml] 18 Mar 2012 Abstract Graphical models use graphs to compactly
More informationInformation Processing Letters
Information Processing Letters 112 (2012) 449 456 Contents lists available at SciVerse ScienceDirect Information Processing Letters www.elsevier.com/locate/ipl Recursive sum product algorithm for generalized
More informationRecitation 4: Elimination algorithm, reconstituted graph, triangulation
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Recitation 4: Elimination algorithm, reconstituted graph, triangulation
More informationFaster parameterized algorithms for Minimum Fill-In
Faster parameterized algorithms for Minimum Fill-In Hans L. Bodlaender Pinar Heggernes Yngve Villanger Technical Report UU-CS-2008-042 December 2008 Department of Information and Computing Sciences Utrecht
More informationCS 343: Artificial Intelligence
CS 343: Artificial Intelligence Bayes Nets: Inference Prof. Scott Niekum The University of Texas at Austin [These slides based on those of Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley.
More informationCollective classification in network data
1 / 50 Collective classification in network data Seminar on graphs, UCSB 2009 Outline 2 / 50 1 Problem 2 Methods Local methods Global methods 3 Experiments Outline 3 / 50 1 Problem 2 Methods Local methods
More informationBayesian Networks Inference (continued) Learning
Learning BN tutorial: ftp://ftp.research.microsoft.com/pub/tr/tr-95-06.pdf TAN paper: http://www.cs.huji.ac.il/~nir/abstracts/frgg1.html Bayesian Networks Inference (continued) Learning Machine Learning
More informationRegularization and Markov Random Fields (MRF) CS 664 Spring 2008
Regularization and Markov Random Fields (MRF) CS 664 Spring 2008 Regularization in Low Level Vision Low level vision problems concerned with estimating some quantity at each pixel Visual motion (u(x,y),v(x,y))
More informationLecture 9: Undirected Graphical Models Machine Learning
Lecture 9: Undirected Graphical Models Machine Learning Andrew Rosenberg March 5, 2010 1/1 Today Graphical Models Probabilities in Undirected Graphs 2/1 Undirected Graphs What if we allow undirected graphs?
More information5 Minimal I-Maps, Chordal Graphs, Trees, and Markov Chains
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms for Inference Fall 2014 5 Minimal I-Maps, Chordal Graphs, Trees, and Markov Chains Recall
More informationLearning Bayesian Networks (part 3) Goals for the lecture
Learning Bayesian Networks (part 3) Mark Craven and David Page Computer Sciences 760 Spring 2018 www.biostat.wisc.edu/~craven/cs760/ Some of the slides in these lectures have been adapted/borrowed from
More information2. Graphical Models. Undirected pairwise graphical models. Factor graphs. Bayesian networks. Conversion between graphical models. Graphical Models 2-1
Graphical Models 2-1 2. Graphical Models Undirected pairwise graphical models Factor graphs Bayesian networks Conversion between graphical models Graphical Models 2-2 Graphical models Families of graphical
More information(67686) Mathematical Foundations of AI July 30, Lecture 11
(67686) Mathematical Foundations of AI July 30, 2008 Lecturer: Ariel D. Procaccia Lecture 11 Scribe: Michael Zuckerman and Na ama Zohary 1 Cooperative Games N = {1,...,n} is the set of players (agents).
More informationAlgorithms for Markov Random Fields in Computer Vision
Algorithms for Markov Random Fields in Computer Vision Dan Huttenlocher November, 2003 (Joint work with Pedro Felzenszwalb) Random Field Broadly applicable stochastic model Collection of n sites S Hidden
More informationAn Introduction to Graph Theory
An Introduction to Graph Theory CIS008-2 Logic and Foundations of Mathematics David Goodwin david.goodwin@perisic.com 12:00, Friday 17 th February 2012 Outline 1 Graphs 2 Paths and cycles 3 Graphs and
More information10708 Graphical Models: Homework 2
10708 Graphical Models: Homework 2 Due October 15th, beginning of class October 1, 2008 Instructions: There are six questions on this assignment. Each question has the name of one of the TAs beside it,
More informationTreewidth and graph minors
Treewidth and graph minors Lectures 9 and 10, December 29, 2011, January 5, 2012 We shall touch upon the theory of Graph Minors by Robertson and Seymour. This theory gives a very general condition under
More informationMarkov Random Fields and Gibbs Sampling for Image Denoising
Markov Random Fields and Gibbs Sampling for Image Denoising Chang Yue Electrical Engineering Stanford University changyue@stanfoed.edu Abstract This project applies Gibbs Sampling based on different Markov
More informationLecture 2 - Introduction to Polytopes
Lecture 2 - Introduction to Polytopes Optimization and Approximation - ENS M1 Nicolas Bousquet 1 Reminder of Linear Algebra definitions Let x 1,..., x m be points in R n and λ 1,..., λ m be real numbers.
More informationBayesian Classification Using Probabilistic Graphical Models
San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 2014 Bayesian Classification Using Probabilistic Graphical Models Mehal Patel San Jose State University
More informationBayesian Networks Inference
Bayesian Networks Inference Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University November 5 th, 2007 2005-2007 Carlos Guestrin 1 General probabilistic inference Flu Allergy Query: Sinus
More informationAlgorithms for Nearest Neighbors
Algorithms for Nearest Neighbors Classic Ideas, New Ideas Yury Lifshits Steklov Institute of Mathematics at St.Petersburg http://logic.pdmi.ras.ru/~yura University of Toronto, July 2007 1 / 39 Outline
More information10 Sum-product on factor tree graphs, MAP elimination
assachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 10 Sum-product on factor tree graphs, AP elimination algorithm The
More informationApplied Mathematics Letters. Graph triangulations and the compatibility of unrooted phylogenetic trees
Applied Mathematics Letters 24 (2011) 719 723 Contents lists available at ScienceDirect Applied Mathematics Letters journal homepage: www.elsevier.com/locate/aml Graph triangulations and the compatibility
More informationSTAT 598L Probabilistic Graphical Models. Instructor: Sergey Kirshner. Exact Inference
STAT 598L Probabilistic Graphical Models Instructor: Sergey Kirshner Exact Inference What To Do With Bayesian/Markov Network? Compact representation of a complex model, but Goal: efficient extraction of
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Raquel Urtasun and Tamir Hazan TTI Chicago April 22, 2011 Raquel Urtasun and Tamir Hazan (TTI-C) Graphical Models April 22, 2011 1 / 22 If the graph is non-chordal, then
More information1) Give decision trees to represent the following Boolean functions:
1) Give decision trees to represent the following Boolean functions: 1) A B 2) A [B C] 3) A XOR B 4) [A B] [C Dl Answer: 1) A B 2) A [B C] 1 3) A XOR B = (A B) ( A B) 4) [A B] [C D] 2 2) Consider the following
More informationLearning decomposable models with a bounded clique size
Learning decomposable models with a bounded clique size Achievements 2014-2016 Aritz Pérez Basque Center for Applied Mathematics Bilbao, March, 2016 Outline 1 Motivation and background 2 The problem 3
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Raquel Urtasun and Tamir Hazan TTI Chicago April 25, 2011 Raquel Urtasun and Tamir Hazan (TTI-C) Graphical Models April 25, 2011 1 / 17 Clique Trees Today we are going to
More informationExact Algorithms Lecture 7: FPT Hardness and the ETH
Exact Algorithms Lecture 7: FPT Hardness and the ETH February 12, 2016 Lecturer: Michael Lampis 1 Reminder: FPT algorithms Definition 1. A parameterized problem is a function from (χ, k) {0, 1} N to {0,
More informationSmall Survey on Perfect Graphs
Small Survey on Perfect Graphs Michele Alberti ENS Lyon December 8, 2010 Abstract This is a small survey on the exciting world of Perfect Graphs. We will see when a graph is perfect and which are families
More informationPart I: Sum Product Algorithm and (Loopy) Belief Propagation. What s wrong with VarElim. Forwards algorithm (filtering) Forwards-backwards algorithm
OU 56 Probabilistic Graphical Models Loopy Belief Propagation and lique Trees / Join Trees lides from Kevin Murphy s Graphical Model Tutorial (with minor changes) eading: Koller and Friedman h 0 Part I:
More informationCS 188: Artificial Intelligence
CS 188: Artificial Intelligence Bayes Nets: Inference Instructors: Dan Klein and Pieter Abbeel --- University of California, Berkeley [These slides were created by Dan Klein and Pieter Abbeel for CS188
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Raquel Urtasun and Tamir Hazan TTI Chicago April 8, 2011 Raquel Urtasun and Tamir Hazan (TTI-C) Graphical Models April 8, 2011 1 / 19 Factor Graphs H does not reveal the
More informationBuilding Classifiers using Bayesian Networks
Building Classifiers using Bayesian Networks Nir Friedman and Moises Goldszmidt 1997 Presented by Brian Collins and Lukas Seitlinger Paper Summary The Naive Bayes classifier has reasonable performance
More informationChordal deletion is fixed-parameter tractable
Chordal deletion is fixed-parameter tractable Dániel Marx Institut für Informatik, Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany. dmarx@informatik.hu-berlin.de Abstract. It
More informationCS 534: Computer Vision Segmentation II Graph Cuts and Image Segmentation
CS 534: Computer Vision Segmentation II Graph Cuts and Image Segmentation Spring 2005 Ahmed Elgammal Dept of Computer Science CS 534 Segmentation II - 1 Outlines What is Graph cuts Graph-based clustering
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Recitation-6: Hardness of Inference Contents 1 NP-Hardness Part-II
More informationGraphs: Introduction. Ali Shokoufandeh, Department of Computer Science, Drexel University
Graphs: Introduction Ali Shokoufandeh, Department of Computer Science, Drexel University Overview of this talk Introduction: Notations and Definitions Graphs and Modeling Algorithmic Graph Theory and Combinatorial
More informationAn Effective Upperbound on Treewidth Using Partial Fill-in of Separators
An Effective Upperbound on Treewidth Using Partial Fill-in of Separators Boi Faltings Martin Charles Golumbic June 28, 2009 Abstract Partitioning a graph using graph separators, and particularly clique
More informationRecap: Gaussian (or Normal) Distribution. Recap: Minimizing the Expected Loss. Topics of This Lecture. Recap: Maximum Likelihood Approach
Truth Course Outline Machine Learning Lecture 3 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Probability Density Estimation II 2.04.205 Discriminative Approaches (5 weeks)
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2016 Lecturer: Trevor Cohn 21. Independence in PGMs; Example PGMs Independence PGMs encode assumption of statistical independence between variables. Critical
More informationEdge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles
Theory and Applications of Graphs Volume 4 Issue 2 Article 2 November 2017 Edge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles Peter Johnson johnspd@auburn.edu Andrew Owens Auburn
More informationDecomposition of log-linear models
Graphical Models, Lecture 5, Michaelmas Term 2009 October 27, 2009 Generating class Dependence graph of log-linear model Conformal graphical models Factor graphs A density f factorizes w.r.t. A if there
More informationMC 302 GRAPH THEORY 10/1/13 Solutions to HW #2 50 points + 6 XC points
MC 0 GRAPH THEORY 0// Solutions to HW # 0 points + XC points ) [CH] p.,..7. This problem introduces an important class of graphs called the hypercubes or k-cubes, Q, Q, Q, etc. I suggest that before you
More informationMatching Algorithms. Proof. If a bipartite graph has a perfect matching, then it is easy to see that the right hand side is a necessary condition.
18.433 Combinatorial Optimization Matching Algorithms September 9,14,16 Lecturer: Santosh Vempala Given a graph G = (V, E), a matching M is a set of edges with the property that no two of the edges have
More informationSum-Product Networks. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 15, 2015
Sum-Product Networks STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 15, 2015 Introduction Outline What is a Sum-Product Network? Inference Applications In more depth
More informationGenerative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 013-014 Jakob Verbeek, December 13+0, 013 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.13.14
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Lecture 8 Junction Trees CS/CNS/EE 155 Andreas Krause Announcements Homework 2 due next Wednesday (Nov 4) in class Start early!!! Project milestones due Monday (Nov 9) 4
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationGraphical Modeling for High Dimensional Data
Journal of Modern Applied Statistical Methods Volume 11 Issue 2 Article 17 11-1-2012 Graphical Modeling for High Dimensional Data Munni Begum Ball State University, Muncie, IN Jay Bagga Ball State University,
More informationMarkov Logic: Representation
Markov Logic: Representation Overview Statistical relational learning Markov logic Basic inference Basic learning Statistical Relational Learning Goals: Combine (subsets of) logic and probability into
More informationModule 7. Independent sets, coverings. and matchings. Contents
Module 7 Independent sets, coverings Contents and matchings 7.1 Introduction.......................... 152 7.2 Independent sets and coverings: basic equations..... 152 7.3 Matchings in bipartite graphs................
More informationK 4 C 5. Figure 4.5: Some well known family of graphs
08 CHAPTER. TOPICS IN CLASSICAL GRAPH THEORY K, K K K, K K, K K, K C C C C 6 6 P P P P P. Graph Operations Figure.: Some well known family of graphs A graph Y = (V,E ) is said to be a subgraph of a graph
More informationFaster parameterized algorithms for Minimum Fill-In
Faster parameterized algorithms for Minimum Fill-In Hans L. Bodlaender Pinar Heggernes Yngve Villanger Abstract We present two parameterized algorithms for the Minimum Fill-In problem, also known as Chordal
More information