Learning the Structure of Deep, Sparse Graphical Models
|
|
|
- Suzanna Williams
- 6 years ago
- Views:
Transcription
1 Learning the Structure of Deep, Sparse Graphical Models Hanna M. Wallach University of Massachusetts Amherst Joint work with Ryan Prescott Adams & Zoubin Ghahramani
2 Deep Belief Networks Deep belief nets are probabilistic generative models that are composed of multiple layers of stochastic latent variables. The latent variables typically have binary values and are often called hidden units or feature detectors. [...] The lower layers receive top-down, directed connections from the layer above. The states of the units in the lowest layer represent a data vector. Geoff Hinton ('09) Scholarpedia Hanna M. Wallach :: UMass Amherst :: 2
3 Network Structure Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 3
4 Network Structure Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 4
5 Network Structure Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 5
6 Network Structure Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 6
7 Network Structure??? Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 7
8 Network Structure Structural questions: # units in each hidden layer? # hidden layers? What network connectivity? What type(s) of unit behavior? Goal: learn the structure Hanna M. Wallach :: UMass Amherst :: 8
9 This Talk A nonparametric Bayesian approach for learning the structure of a layered, directed, deep belief network. + [Mike Jordan] [Yee Whye Teh] [Geoff Hinton] [Zoubin Ghahramani] Hanna M. Wallach :: UMass Amherst :: 9
10 Outline Background: finite single-layer networks Infinite belief networks: Learning the number of hidden units in each layer Learning the number of hidden layers Learning the type(s) of unit behavior Experimental results Hanna M. Wallach :: UMass Amherst :: 10
11 Outline Background: finite single-layer networks Hanna M. Wallach :: UMass Amherst :: 11
12 Finite Single-Layer Networks hiddens visibles Use a binary matrix to represent the edge structure (connectivity) of a directed graph A prior distribution on binary matrices a prior distribution on singlelayered belief networks Hanna M. Wallach :: UMass Amherst :: 12
13 Finite Single-Layer Networks hiddens visibles An infinite number of columns an infinite number of hidden units Can we let these binary matrices to have an infinite number of columns? Yes: Indian buffet process Hanna M. Wallach :: UMass Amherst :: 13
14 Outline Background: finite single-layer networks Infinite belief networks: Learning the number of hidden units in each layer Hanna M. Wallach :: UMass Amherst :: 14
15 Infinitely-Wide Layers [Wood, Griffiths & Ghahramani, '06] Use an Indian buffet process (IBP) as a prior on binary matrices with countably infinite columns: Unbounded number of hidden units Posterior inference determines the subset of hidden units responsible for the observations The IBP ensures that the matrices are extremely sparse: always a finite number of nonzero columns Finite number of actively-used hidden units Hanna M. Wallach :: UMass Amherst :: 15
16 The Indian Buffet Process [Ghahramani et al., '07] customers dishes Parameters: α and β First customer tries Poisson(α) dishes Hanna M. Wallach :: UMass Amherst :: 16
17 The Indian Buffet Process [Ghahramani et al., '07] customers dishes Parameters: α and β First customer tries Poisson(α) dishes n th customer tries: Previously-tasted dish k with probability nk / (β + n - 1) Poisson(αβ / (β + n 1)) completely new dishes Hanna M. Wallach :: UMass Amherst :: 17
18 The Indian Buffet Process [Ghahramani et al., '07] customers dishes Parameters: α and β First customer tries Poisson(α) dishes n th customer tries: Previously-tasted dish k with probability nk / (β + n - 1) Poisson(αβ / (β + n 1)) completely new dishes Hanna M. Wallach :: UMass Amherst :: 18
19 The Indian Buffet Process [Ghahramani et al., '07] customers dishes Parameters: α and β First customer tries Poisson(α) dishes n th customer tries: Previously-tasted dish k with probability nk / (β + n - 1) Poisson(αβ / (β + n 1)) completely new dishes Hanna M. Wallach :: UMass Amherst :: 19
20 Properties of the IBP For a finite number of customers, there will always be a finite number of dishes tasted Infinitely exchangeable rows and columns There is a related stick-breaking construction Popular for shared latent feature (hidden cause) models Latent features can be added/removed from a model without dimensionality-altering MCMC methods Hanna M. Wallach :: UMass Amherst :: 20
21 Single-Layer Belief Networks [Wood, Griffiths & Ghahramani, '06] customers dishes Parameters: α and β... customers = observed units dishes = hidden units Hanna M. Wallach :: UMass Amherst :: 21
22 Single-Layer Belief Networks [Wood, Griffiths & Ghahramani, '06] customers dishes Parameters: α and β... customers = observed units dishes = hidden units Hanna M. Wallach :: UMass Amherst :: 22
23 Single-Layer Belief Networks [Wood, Griffiths & Ghahramani, '06] customers dishes Parameters: α and β... customers = observed units dishes = hidden units Hanna M. Wallach :: UMass Amherst :: 23
24 Single-Layer Belief Networks [Wood, Griffiths & Ghahramani, '06] customers dishes Parameters: α and β... customers = observed units dishes = hidden units Hanna M. Wallach :: UMass Amherst :: 24
25 Single-Layer Multi-Layer? Single-layer belief networks have limited utility: Hidden units are independent a priori Deep networks = multiple hidden layers: Hidden units are dependent a priori Goal: extend the IBP in order to construct deep belief networks with unbounded width and depth Hanna M. Wallach :: UMass Amherst :: 25
26 Outline Background: finite single-layer networks Infinite belief networks: Learning the number of hidden units in each layer Learning the number of hidden layers Hanna M. Wallach :: UMass Amherst :: 26
27 Multi-Layer Belief Networks... Hanna M. Wallach :: UMass Amherst :: 27
28 Multi-Layer Belief Networks Hanna M. Wallach :: UMass Amherst :: 28
29 Multi-Layer Belief Networks Hanna M. Wallach :: UMass Amherst :: 29
30 Multi-Layer Belief Networks Hanna M. Wallach :: UMass Amherst :: 30
31 Layered Belief Nets We could use a finite number of IBPs But... what about using an infinite recursion: Every dish is a customer in another restaurant Unbounded number of layers, each of unbounded width Remarkably, we will always hit a layer with zero units! Always stop at a finite but unbounded depth We don't have to make an ad hoc choice of depth Hanna M. Wallach :: UMass Amherst :: 31
32 The Cascading IBP (CIBP) A stochastic process which results in an infinite sequence of infinite binary matrices Each matrix is exchangeable in both rows and columns How do we know the CIBP converges? The number of dishes in one layer depends only on the number of customers in the previous layer Can prove that this Markov chain reaches an absorbing state in finite time with probability one Hanna M. Wallach :: UMass Amherst :: 32
33 CIBP Properties For a unit in layer m+1: Expected # of parents: α Expected # of children: c(β, Km) = 1 + (Km - 1) / (1 + β) lim β 0 c(β, Km) = Km and lim β c(β, Km) = 1 We do not want network properties to be constant at all depths, e.g., some levels should be sparser than others: Each layer can have different IBP parameters α and β so long as they are bounded from above Hanna M. Wallach :: UMass Amherst :: 33
34 Samples from the CIBP Prior Hanna M. Wallach :: UMass Amherst :: 34
35 Outline Background: finite single-layer networks Infinite belief networks: Learning the number of hidden units in each layer Learning the number of hidden layers Learning the type(s) of unit behavior Hanna M. Wallach :: UMass Amherst :: 35
36 Learning Unit Behavior [Frey, '97; Frey & Hinton, '99] Unit activations are weighted linear sums with biases Nonlinear Gaussian belief network approach: sigmoid(activation + Gaussian noise with precision ν) Hanna M. Wallach :: UMass Amherst :: 36
37 Priors and Inference Layer-wise Gaussian priors on weights and biases Layer-wise Gamma priors on noise precisions Layer-wise parameters tied via global hyperparameters Markov chain Monte Carlo inference: Parameter inference is easy given the network structure Edges added/removed using Metropolis-Hastings Hanna M. Wallach :: UMass Amherst :: 37
38 Outline Background: finite single-layer networks Infinite belief networks: Learning the number of hidden units in each layer Learning the number of hidden layers Learning the type(s) of unit behavior Experimental results Hanna M. Wallach :: UMass Amherst :: 38
39 Experimental Results Olivetti Faces: images of 40 faces; 64x64 Inferred: ~3 hidden layers; 70 units per hidden layer MNIST Digits: images of 10 digits; 28x28 Inferred: ~3 hidden layers; 120, 100, 70 units Frey Faces: images of Brendan Frey; 20x28 Inferred: ~3 hidden layers; 260, 120, 35 units Hanna M. Wallach :: UMass Amherst :: 39
40 Olivetti: Reconstructions & Features Hanna M. Wallach :: UMass Amherst :: 40
41 Olivetti: Fantasies & Activations Hanna M. Wallach :: UMass Amherst :: 41
42 MNIST Digits Hanna M. Wallach :: UMass Amherst :: 42
43 Frey Faces Hanna M. Wallach :: UMass Amherst :: 43
44 Summary United deep belief networks & Bayesian nonparametrics Introduced the CIBP & proved convergence properties Addressed 3 issues with deep belief networks: Number of units in each hidden layer Number of hidden layers Type(s) of hidden unit behavior Hanna M. Wallach :: UMass Amherst :: 44
45 Thanks! Ryan Prescott Adams & Zoubin Ghahramani Acknowledgements: Brendan Frey, David MacKay, Iain Murray, Radford Neal
Modeling Dyadic Data with Binary Latent Factors
Modeling Dyadic Data with Binary Latent Factors Edward Meeds Department of Computer Science University of Toronto ewm@cs.toronto.edu Radford Neal Department of Computer Science University of Toronto radford@cs.toronto.edu
Learning Global-to-Local Discrete Components with Nonparametric Bayesian Feature Construction
Learning Global-to-Local Discrete Components with Nonparametric Bayesian Feature Construction Min-Oh Heo Seoul, 151-742, Korea moheo@bi.snu.ac.kr Jaeseon Lee Field Robot R&D Group Korea Institute of Industrial
A Fast Learning Algorithm for Deep Belief Nets
A Fast Learning Algorithm for Deep Belief Nets Geoffrey E. Hinton, Simon Osindero Department of Computer Science University of Toronto, Toronto, Canada Yee-Whye Teh Department of Computer Science National
Learning Objects and Parts in Images
Learning Objects and Parts in Images Chris Williams E H U N I V E R S I T T Y O H F R G E D I N B U School of Informatics, University of Edinburgh, UK Learning multiple objects and parts from images (joint
Probabilistic Graphical Models
Overview of Part Two Probabilistic Graphical Models Part Two: Inference and Learning Christopher M. Bishop Exact inference and the junction tree MCMC Variational methods and EM Example General variational
Beam Search based MAP Estimates for the Indian Buffet Process
Piyush Rai School of Computing, University of Utah, Salt Lake City, UT, USA Hal Daumé III Department of Computer Science, University of Maryland, College Park, MD, USA piyush@cs.utah.edu hal@umiacs.umd.edu
Large Scale Nonparametric Bayesian Inference: Data Parallelisation in the Indian Buffet Process
Large Scale Nonparametric Bayesian Inference: Data Parallelisation in the Indian Buffet Process Finale Doshi-Velez University of Cambridge Cambridge, CB2PZ, UK finale@alum.mit.edu David Knowles University
Latent Regression Bayesian Network for Data Representation
2016 23rd International Conference on Pattern Recognition (ICPR) Cancún Center, Cancún, México, December 4-8, 2016 Latent Regression Bayesian Network for Data Representation Siqi Nie Department of Electrical,
A Nonparametric Bayesian Approach to Modeling Overlapping Clusters
A Nonparametric Bayesian Approach to Modeling Overlapping Clusters Katherine A. Heller Gatsby Computational Neuroscience Unit University College London Zoubin Ghahramani Engineering Department University
CS281 Section 9: Graph Models and Practical MCMC
CS281 Section 9: Graph Models and Practical MCMC Scott Linderman November 11, 213 Now that we have a few MCMC inference algorithms in our toolbox, let s try them out on some random graph models. Graphs
Keeping flexible active contours on track using Metropolis updates
Keeping flexible active contours on track using Metropolis updates Trausti T. Kristjansson University of Waterloo ttkri stj @uwater l oo. ca Brendan J. Frey University of Waterloo f r ey@uwater l oo. ca
Claims Severity Modeling. Radhika Anand. Program in Statistical and Economic Modeling Duke University. Date: Approved. Sayan Mukherjee, Supervisor
Claims Severity Modeling by Radhika Anand Program in Statistical and Economic Modeling Duke University Date: Approved Sayan Mukherjee, Supervisor Surya Tapas Tokdar Kent P Kimbrough Thesis submitted in
Short-Cut MCMC: An Alternative to Adaptation
Short-Cut MCMC: An Alternative to Adaptation Radford M. Neal Dept. of Statistics and Dept. of Computer Science University of Toronto http://www.cs.utoronto.ca/ radford/ Third Workshop on Monte Carlo Methods,
Non-Parametric Bayesian Sum-Product Networks
Sang-Woo Lee slee@bi.snu.ac.kr School of Computer Sci. and Eng., Seoul National University, Seoul 151-744, Korea Christopher J. Watkins c.j.watkins@rhul.ac.uk Department of Computer Science, Royal Holloway,
Variational Methods for Graphical Models
Chapter 2 Variational Methods for Graphical Models 2.1 Introduction The problem of probabb1istic inference in graphical models is the problem of computing a conditional probability distribution over the
Deep Boltzmann Machines
Deep Boltzmann Machines Sargur N. Srihari srihari@cedar.buffalo.edu Topics 1. Boltzmann machines 2. Restricted Boltzmann machines 3. Deep Belief Networks 4. Deep Boltzmann machines 5. Boltzmann machines
A Nonparametric Bayesian Approach to Detecting Spatial Activation Patterns in fmri Data
A Nonparametric Bayesian Approach to Detecting Spatial Activation Patterns in fmri Data Seyoung Kim, Padhraic Smyth, and Hal Stern Bren School of Information and Computer Sciences University of California,
Approximate Bayesian Computation. Alireza Shafaei - April 2016
Approximate Bayesian Computation Alireza Shafaei - April 2016 The Problem Given a dataset, we are interested in. The Problem Given a dataset, we are interested in. The Problem Given a dataset, we are interested
Computer vision: models, learning and inference. Chapter 10 Graphical Models
Computer vision: models, learning and inference Chapter 10 Graphical Models Independence Two variables x 1 and x 2 are independent if their joint probability distribution factorizes as Pr(x 1, x 2 )=Pr(x
Geoff McLachlan and Angus Ng. University of Queensland. Schlumberger Chaired Professor Univ. of Texas at Austin. + Chris Bishop
EM Algorithm Geoff McLachlan and Angus Ng Department of Mathematics & Institute for Molecular Bioscience University of Queensland Adapted by Joydeep Ghosh Schlumberger Chaired Professor Univ. of Texas
Particle Filters for Visual Tracking
Particle Filters for Visual Tracking T. Chateau, Pascal Institute, Clermont-Ferrand 1 Content Particle filtering: a probabilistic framework SIR particle filter MCMC particle filter RJMCMC particle filter
Reddit Recommendation System Daniel Poon, Yu Wu, David (Qifan) Zhang CS229, Stanford University December 11 th, 2011
Reddit Recommendation System Daniel Poon, Yu Wu, David (Qifan) Zhang CS229, Stanford University December 11 th, 2011 1. Introduction Reddit is one of the most popular online social news websites with millions
Convexization in Markov Chain Monte Carlo
in Markov Chain Monte Carlo 1 IBM T. J. Watson Yorktown Heights, NY 2 Department of Aerospace Engineering Technion, Israel August 23, 2011 Problem Statement MCMC processes in general are governed by non
Nested Sampling: Introduction and Implementation
UNIVERSITY OF TEXAS AT SAN ANTONIO Nested Sampling: Introduction and Implementation Liang Jing May 2009 1 1 ABSTRACT Nested Sampling is a new technique to calculate the evidence, Z = P(D M) = p(d θ, M)p(θ
Towards Asynchronous Distributed MCMC Inference for Large Graphical Models
Towards Asynchronous Distributed MCMC for Large Graphical Models Sameer Singh Department of Computer Science University of Massachusetts Amherst, MA 13 sameer@cs.umass.edu Andrew McCallum Department of
A Dynamic Relational Infinite Feature Model for Longitudinal Social Networks
A Dynamic Relational Infinite Feature Model for Longitudinal Social Networs James Foulds Christopher DuBois Arthur U. Asuncion Carter T. Butts Padhraic Smyth Department of Computer Science University of
José Miguel Hernández Lobato Zoubin Ghahramani Computational and Biological Learning Laboratory Cambridge University
José Miguel Hernández Lobato Zoubin Ghahramani Computational and Biological Learning Laboratory Cambridge University 20/09/2011 1 Evaluation of data mining and machine learning methods in the task of modeling
Ludwig Fahrmeir Gerhard Tute. Statistical odelling Based on Generalized Linear Model. íecond Edition. . Springer
Ludwig Fahrmeir Gerhard Tute Statistical odelling Based on Generalized Linear Model íecond Edition. Springer Preface to the Second Edition Preface to the First Edition List of Examples List of Figures
ECE521: Week 11, Lecture March 2017: HMM learning/inference. With thanks to Russ Salakhutdinov
ECE521: Week 11, Lecture 20 27 March 2017: HMM learning/inference With thanks to Russ Salakhutdinov Examples of other perspectives Murphy 17.4 End of Russell & Norvig 15.2 (Artificial Intelligence: A Modern
Computer Vision Group Prof. Daniel Cremers. 4. Probabilistic Graphical Models Directed Models
Prof. Daniel Cremers 4. Probabilistic Graphical Models Directed Models The Bayes Filter (Rep.) (Bayes) (Markov) (Tot. prob.) (Markov) (Markov) 2 Graphical Representation (Rep.) We can describe the overall
Semi-Supervised Learning with Trees
Semi-Supervised Learning with Trees Charles Kemp, Thomas L. Griffiths, Sean Stromsten & Joshua B. Tenenbaum Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 0139 {ckemp,gruffydd,sean s,jbt}@mit.edu
Statistical Matching using Fractional Imputation
Statistical Matching using Fractional Imputation Jae-Kwang Kim 1 Iowa State University 1 Joint work with Emily Berg and Taesung Park 1 Introduction 2 Classical Approaches 3 Proposed method 4 Application:
Loopy Belief Propagation
Loopy Belief Propagation Research Exam Kristin Branson September 29, 2003 Loopy Belief Propagation p.1/73 Problem Formalization Reasoning about any real-world problem requires assumptions about the structure
Post-Processing for MCMC
ost-rocessing for MCMC Edwin D. de Jong Marco A. Wiering Mădălina M. Drugan institute of information and computing sciences, utrecht university technical report UU-CS-23-2 www.cs.uu.nl ost-rocessing for
Lecture 13. Deep Belief Networks. Michael Picheny, Bhuvana Ramabhadran, Stanley F. Chen
Lecture 13 Deep Belief Networks Michael Picheny, Bhuvana Ramabhadran, Stanley F. Chen IBM T.J. Watson Research Center Yorktown Heights, New York, USA {picheny,bhuvana,stanchen}@us.ibm.com 12 December 2012
Introduction to Generative Adversarial Networks
Introduction to Generative Adversarial Networks Luke de Oliveira Vai Technologies Lawrence Berkeley National Laboratory @lukede0 @lukedeo lukedeo@vaitech.io https://ldo.io 1 Outline Why Generative Modeling?
Variational Autoencoders. Sargur N. Srihari
Variational Autoencoders Sargur N. srihari@cedar.buffalo.edu Topics 1. Generative Model 2. Standard Autoencoder 3. Variational autoencoders (VAE) 2 Generative Model A variational autoencoder (VAE) is a
Efficient Feature Learning Using Perturb-and-MAP
Efficient Feature Learning Using Perturb-and-MAP Ke Li, Kevin Swersky, Richard Zemel Dept. of Computer Science, University of Toronto {keli,kswersky,zemel}@cs.toronto.edu Abstract Perturb-and-MAP [1] is
CSC412: Stochastic Variational Inference. David Duvenaud
CSC412: Stochastic Variational Inference David Duvenaud Admin A3 will be released this week and will be shorter Motivation for REINFORCE Class projects Class Project ideas Develop a generative model for
Markov Random Fields and Gibbs Sampling for Image Denoising
Markov Random Fields and Gibbs Sampling for Image Denoising Chang Yue Electrical Engineering Stanford University changyue@stanfoed.edu Abstract This project applies Gibbs Sampling based on different Markov
Computer Vision Group Prof. Daniel Cremers. 4. Probabilistic Graphical Models Directed Models
Prof. Daniel Cremers 4. Probabilistic Graphical Models Directed Models The Bayes Filter (Rep.) (Bayes) (Markov) (Tot. prob.) (Markov) (Markov) 2 Graphical Representation (Rep.) We can describe the overall
3. Replace any row by the sum of that row and a constant multiple of any other row.
Math Section. Section.: Solving Systems of Linear Equations Using Matrices As you may recall from College Algebra or Section., you can solve a system of linear equations in two variables easily by applying
D-Separation. b) the arrows meet head-to-head at the node, and neither the node, nor any of its descendants, are in the set C.
D-Separation Say: A, B, and C are non-intersecting subsets of nodes in a directed graph. A path from A to B is blocked by C if it contains a node such that either a) the arrows on the path meet either
Stacked Denoising Autoencoders for Face Pose Normalization
Stacked Denoising Autoencoders for Face Pose Normalization Yoonseop Kang 1, Kang-Tae Lee 2,JihyunEun 2, Sung Eun Park 2 and Seungjin Choi 1 1 Department of Computer Science and Engineering Pohang University
Bayesian Spatiotemporal Modeling with Hierarchical Spatial Priors for fmri
Bayesian Spatiotemporal Modeling with Hierarchical Spatial Priors for fmri Galin L. Jones 1 School of Statistics University of Minnesota March 2015 1 Joint with Martin Bezener and John Hughes Experiment
18 October, 2013 MVA ENS Cachan. Lecture 6: Introduction to graphical models Iasonas Kokkinos
Machine Learning for Computer Vision 1 18 October, 2013 MVA ENS Cachan Lecture 6: Introduction to graphical models Iasonas Kokkinos Iasonas.kokkinos@ecp.fr Center for Visual Computing Ecole Centrale Paris
RJaCGH, a package for analysis of
RJaCGH, a package for analysis of CGH arrays with Reversible Jump MCMC 1. CGH Arrays: Biological problem: Changes in number of DNA copies are associated to cancer activity. Microarray technology: Oscar
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Bayesian Networks Directed Acyclic Graph (DAG) Bayesian Networks General Factorization Bayesian Curve Fitting (1) Polynomial Bayesian
Learning stick-figure models using nonparametric Bayesian priors over trees
Learning stick-figure models using nonparametric Bayesian priors over trees Edward W. Meeds, David A. Ross, Richard S. Zemel, and Sam T. Roweis Department of Computer Science University of Toronto {ewm,
Clustering Relational Data using the Infinite Relational Model
Clustering Relational Data using the Infinite Relational Model Ana Daglis Supervised by: Matthew Ludkin September 4, 2015 Ana Daglis Clustering Data using the Infinite Relational Model September 4, 2015
A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process
A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process Bahadir Ozdemir Department of Computer Science University of Maryland College Park, MD 2742 USA ozdemir@cs.umd.edu
Unsupervised Discovery of Non-Linear Structure using Contrastive Backpropagation
Unsupervised Discovery of Non-Linear Structure using Contrastive Backpropagation G. E. Hinton, S. Osindero, M. Welling and Y. W. Teh Department of Computer Science University of Toronto Toronto, Canada
Modified Metropolis-Hastings algorithm with delayed rejection
Modified Metropolis-Hastings algorithm with delayed reection K.M. Zuev & L.S. Katafygiotis Department of Civil Engineering, Hong Kong University of Science and Technology, Hong Kong, China ABSTRACT: The
Alternatives to Direct Supervision
CreativeAI: Deep Learning for Graphics Alternatives to Direct Supervision Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Theory and Basics State of
Scene Grammars, Factor Graphs, and Belief Propagation
Scene Grammars, Factor Graphs, and Belief Propagation Pedro Felzenszwalb Brown University Joint work with Jeroen Chua Probabilistic Scene Grammars General purpose framework for image understanding and
Learning Sparse Topographic Representations with Products of Student-t Distributions
Learning Sparse Topographic Representations with Products of Student-t Distributions Max Welling and Geoffrey Hinton Department of Computer Science University of Toronto 10 King s College Road Toronto,
Probabilistic Robotics
Probabilistic Robotics Sebastian Thrun Wolfram Burgard Dieter Fox The MIT Press Cambridge, Massachusetts London, England Preface xvii Acknowledgments xix I Basics 1 1 Introduction 3 1.1 Uncertainty in
Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees
Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees Joseph Gonzalez Yucheng Low Arthur Gretton Carlos Guestrin Draw Samples Sampling as an Inference Procedure Suppose we wanted to know the
1 Methods for Posterior Simulation
1 Methods for Posterior Simulation Let p(θ y) be the posterior. simulation. Koop presents four methods for (posterior) 1. Monte Carlo integration: draw from p(θ y). 2. Gibbs sampler: sequentially drawing
Overview. Monte Carlo Methods. Statistics & Bayesian Inference Lecture 3. Situation At End Of Last Week
Statistics & Bayesian Inference Lecture 3 Joe Zuntz Overview Overview & Motivation Metropolis Hastings Monte Carlo Methods Importance sampling Direct sampling Gibbs sampling Monte-Carlo Markov Chains Emcee
08 An Introduction to Dense Continuous Robotic Mapping
NAVARCH/EECS 568, ROB 530 - Winter 2018 08 An Introduction to Dense Continuous Robotic Mapping Maani Ghaffari March 14, 2018 Previously: Occupancy Grid Maps Pose SLAM graph and its associated dense occupancy
Probabilistic Graphical Models
Probabilistic Graphical Models Lecture 17 EM CS/CNS/EE 155 Andreas Krause Announcements Project poster session on Thursday Dec 3, 4-6pm in Annenberg 2 nd floor atrium! Easels, poster boards and cookies
Akarsh Pokkunuru EECS Department Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Akarsh Pokkunuru EECS Department 03-16-2017 Contractive Auto-Encoders: Explicit Invariance During Feature Extraction 1 AGENDA Introduction to Auto-encoders Types of Auto-encoders Analysis of different
CS242: Probabilistic Graphical Models Lecture 3: Factor Graphs & Variable Elimination
CS242: Probabilistic Graphical Models Lecture 3: Factor Graphs & Variable Elimination Instructor: Erik Sudderth Brown University Computer Science September 11, 2014 Some figures and materials courtesy
Max-Product Particle Belief Propagation
Max-Product Particle Belief Propagation Rajkumar Kothapa Department of Computer Science Brown University Providence, RI 95 rjkumar@cs.brown.edu Jason Pacheco Department of Computer Science Brown University
Day 3 Lecture 1. Unsupervised Learning
Day 3 Lecture 1 Unsupervised Learning Semi-supervised and transfer learning Myth: you can t do deep learning unless you have a million labelled examples for your problem. Reality You can learn useful representations
Graphical Models, Bayesian Method, Sampling, and Variational Inference
Graphical Models, Bayesian Method, Sampling, and Variational Inference With Application in Function MRI Analysis and Other Imaging Problems Wei Liu Scientific Computing and Imaging Institute University
arxiv: v1 [stat.ml] 22 Mar 2014
![arxiv: v1 [stat.ml] 22 Mar 2014](/thumbs/75/72075010.jpg "arxiv: v1 [stat.ml] 22 Mar 2014")
FIREFLY MONTE CARLO: EXACT MCMC WITH SUBSETS OF DATA By Dougal Maclaurin and Ryan P. Adams Harvard University arxiv:1403.5693v1 [stat.ml] 22 Mar 2014 Markov chain Monte Carlo (MCMC) is a popular and successful
CSCI 599 Class Presenta/on. Zach Levine. Markov Chain Monte Carlo (MCMC) HMM Parameter Es/mates
CSCI 599 Class Presenta/on Zach Levine Markov Chain Monte Carlo (MCMC) HMM Parameter Es/mates April 26 th, 2012 Topics Covered in this Presenta2on A (Brief) Review of HMMs HMM Parameter Learning Expecta2on-
Introduction to Graphical Models
Robert Collins CSE586 Introduction to Graphical Models Readings in Prince textbook: Chapters 10 and 11 but mainly only on directed graphs at this time Credits: Several slides are from: Review: Probability
Edge-exchangeable graphs and sparsity
Edge-exchangeable graphs and sparsity Tamara Broderick Department of EECS Massachusetts Institute of Technology tbroderick@csail.mit.edu Diana Cai Department of Statistics University of Chicago dcai@uchicago.edu
Probabilistic Graphical Models
10-708 Probabilistic Graphical Models Homework 4 Due Apr 27, 12:00 noon Submission: Homework is due on the due date at 12:00 noon. Please see course website for policy on late submission. You must submit
GiRaF: a toolbox for Gibbs Random Fields analysis
GiRaF: a toolbox for Gibbs Random Fields analysis Julien Stoehr *1, Pierre Pudlo 2, and Nial Friel 1 1 University College Dublin 2 Aix-Marseille Université February 24, 2016 Abstract GiRaF package offers
arxiv: v1 [cs.lg] 24 Jan 2019
![arxiv: v1 [cs.lg] 24 Jan 2019](/thumbs/88/115092711.jpg "arxiv: v1 [cs.lg] 24 Jan 2019")
Jaehoon Cha Kyeong Soo Kim Sanghuyk Lee arxiv:9.879v [cs.lg] Jan 9 Abstract Noting the importance of the latent variables in inference and learning, we propose a novel framework for autoencoders based
A Deterministic Global Optimization Method for Variational Inference
A Deterministic Global Optimization Method for Variational Inference Hachem Saddiki Mathematics and Statistics University of Massachusetts, Amherst saddiki@math.umass.edu Andrew C. Trapp Operations and
MSA101/MVE Lecture 5
MSA101/MVE187 2017 Lecture 5 Petter Mostad Chalmers University September 12, 2017 1 / 15 Importance sampling MC integration computes h(x)f (x) dx where f (x) is a probability density function, by simulating
Mesh segmentation. Florent Lafarge Inria Sophia Antipolis - Mediterranee
Mesh segmentation Florent Lafarge Inria Sophia Antipolis - Mediterranee Outline What is mesh segmentation? M = {V,E,F} is a mesh S is either V, E or F (usually F) A Segmentation is a set of sub-meshes
Training Restricted Boltzmann Machines with Multi-Tempering: Harnessing Parallelization
Training Restricted Boltzmann Machines with Multi-Tempering: Harnessing Parallelization Philemon Brakel, Sander Dieleman, and Benjamin Schrauwen Department of Electronics and Information Systems, Ghent
Scalable Inference in Hierarchical Generative Models
Scalable Inference in Hierarchical Generative Models Thomas Dean Department of Computer Science Brown University, Providence, RI 02912 Abstract Borrowing insights from computational neuroscience, we present
Scene Grammars, Factor Graphs, and Belief Propagation
Scene Grammars, Factor Graphs, and Belief Propagation Pedro Felzenszwalb Brown University Joint work with Jeroen Chua Probabilistic Scene Grammars General purpose framework for image understanding and
ADAPTIVE METROPOLIS-HASTINGS SAMPLING, OR MONTE CARLO KERNEL ESTIMATION
ADAPTIVE METROPOLIS-HASTINGS SAMPLING, OR MONTE CARLO KERNEL ESTIMATION CHRISTOPHER A. SIMS Abstract. A new algorithm for sampling from an arbitrary pdf. 1. Introduction Consider the standard problem of
Bayesian model ensembling using meta-trained recurrent neural networks
Bayesian model ensembling using meta-trained recurrent neural networks Luca Ambrogioni l.ambrogioni@donders.ru.nl Umut Güçlü u.guclu@donders.ru.nl Yağmur Güçlütürk y.gucluturk@donders.ru.nl Julia Berezutskaya
Automatic Alignment of Local Representations
Automatic Alignment of Local Representations Yee Whye Teh and Sam Roweis Department of Computer Science, University of Toronto ywteh,roweis @cs.toronto.edu Abstract We present an automatic alignment procedure
Nonparametric Bayesian Texture Learning and Synthesis
Appears in Advances in Neural Information Processing Systems (NIPS) 2009. Nonparametric Bayesian Texture Learning and Synthesis Long (Leo) Zhu 1 Yuanhao Chen 2 William Freeman 1 Antonio Torralba 1 1 CSAIL,
STA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 5 Inference
Conditional Random Fields and beyond D A N I E L K H A S H A B I C S U I U C,
Conditional Random Fields and beyond D A N I E L K H A S H A B I C S 5 4 6 U I U C, 2 0 1 3 Outline Modeling Inference Training Applications Outline Modeling Problem definition Discriminative vs. Generative
Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient
Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient Tijmen Tieleman tijmen@cs.toronto.edu Department of Computer Science, University of Toronto, Toronto, Ontario M5S
Summary: A Tutorial on Learning With Bayesian Networks
Summary: A Tutorial on Learning With Bayesian Networks Markus Kalisch May 5, 2006 We primarily summarize [4]. When we think that it is appropriate, we comment on additional facts and more recent developments.
Warped Mixture Models
Warped Mixture Models Tomoharu Iwata, David Duvenaud, Zoubin Ghahramani Cambridge University Computational and Biological Learning Lab March 11, 2013 OUTLINE Motivation Gaussian Process Latent Variable
Calibration and emulation of TIE-GCM
Calibration and emulation of TIE-GCM Serge Guillas School of Mathematics Georgia Institute of Technology Jonathan Rougier University of Bristol Big Thanks to Crystal Linkletter (SFU-SAMSI summer school)
Stochastic Blockmodel with Cluster Overlap, Relevance Selection, and Similarity-Based Smoothing
Stochastic Blockmodel with Cluster Overlap, Relevance Selection, and Similarity-Based Smoothing Joyce Jiyoung Whang, Piyush Rai, and Inderjit S. Dhillon Department of Computer Science The University of
Quantitative Biology II!
Quantitative Biology II! Lecture 3: Markov Chain Monte Carlo! March 9, 2015! 2! Plan for Today!! Introduction to Sampling!! Introduction to MCMC!! Metropolis Algorithm!! Metropolis-Hastings Algorithm!!
Bayesian Networks. A Bayesian network is a directed acyclic graph that represents causal relationships between random variables. Earthquake.
Bayes Nets Independence With joint probability distributions we can compute many useful things, but working with joint PD's is often intractable. The naïve Bayes' approach represents one (boneheaded?)
One-Shot Learning with a Hierarchical Nonparametric Bayesian Model
One-Shot Learning with a Hierarchical Nonparametric Bayesian Model R. Salakhutdinov, J. Tenenbaum and A. Torralba MIT Technical Report, 2010 Presented by Esther Salazar Duke University June 10, 2011 E.
Bilevel Sparse Coding
Adobe Research 345 Park Ave, San Jose, CA Mar 15, 2013 Outline 1 2 The learning model The learning algorithm 3 4 Sparse Modeling Many types of sensory data, e.g., images and audio, are in high-dimensional
Deep Learning. Architecture Design for. Sargur N. Srihari
Architecture Design for Deep Learning Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation
Statistical Techniques in Robotics (16-831, F12) Lecture#05 (Wednesday, September 12) Mapping
Statistical Techniques in Robotics (16-831, F12) Lecture#05 (Wednesday, September 12) Mapping Lecturer: Alex Styler (in for Drew Bagnell) Scribe: Victor Hwang 1 1 Occupancy Mapping When solving the localization
Stat 547 Assignment 3
Stat 547 Assignment 3 Release Date: Saturday April 16, 2011 Due Date: Wednesday, April 27, 2011 at 4:30 PST Note that the deadline for this assignment is one day before the final project deadline, and
From the Jungle to the Garden: Growing Trees for Markov Chain Monte Carlo Inference in Undirected Graphical Models
From the Jungle to the Garden: Growing Trees for Markov Chain Monte Carlo Inference in Undirected Graphical Models by Jean-Noël Rivasseau, M.Sc., Ecole Polytechnique, 2003 A THESIS SUBMITTED IN PARTIAL
Deep Learning With Noise
Deep Learning With Noise Yixin Luo Computer Science Department Carnegie Mellon University yixinluo@cs.cmu.edu Fan Yang Department of Mathematical Sciences Carnegie Mellon University fanyang1@andrew.cmu.edu