Differentiable Inductive Logic Programming Richard Evans,
|
|
|
- Lorin Atkins
- 5 years ago
- Views:
Transcription
1 Differentiable Inductive Logic Programming Richard Evans,
2 Overview Our system, ILP, learns logic programs from examples. ILP learns by back-propagation. It is robust to noisy and ambiguous data.
3 Overview Background ILP Experiments
4 Learning Procedures from Examples Given some input / output examples, learn a general procedure for transforming inputs into outputs.
5 Learning Procedures from Examples Given some input / output examples, learn a general procedure for transforming inputs into outputs.
6 Learning Procedures from Examples Given some input / output examples, learn a general procedure for transforming inputs into outputs.
7 Learning Procedures from Examples We shall consider three approaches: 1. Symbolic program synthesis 2. Neural program induction 3. Neural program synthesis
8 Symbolic Program Synthesis (SPS) Given some input/output examples, they produce an explicit human-readable program that, when evaluated on the inputs, produces the outputs. They use a symbolic search procedure to find the program.
9 Symbolic Program Synthesis (SPS) Input / Output Examples Explicit Program map (reverse. tail. reverse) def remove_last(x): return [y[0:len(y)-1] for y in x]
10 Symbolic Program Synthesis (SPS) Input / Output Examples Explicit Program map (reverse. tail. reverse) def remove_last(x): return [y[0:len(y)-1] for y in x] Examples: MagicHaskeller, λ², Igor-2, Progol, Metagol
11 Symbolic Program Synthesis (SPS) Data-efficient? Yes Interpretable? Yes Generalises outside training data? Yes Robust to mislabelled data? Sometimes Robust to ambiguous data? No
12 Ambiguous Data
13 Neural Program Induction (NPI) Given input/output pairs, a neural network learns a procedure for mapping inputs to outputs. The network generates the output from the input directly, using a latent representation of the program. Here, the general procedure is implicit in the weights of the model.
14 Neural Program Induction (NPI) Examples: Differentiable Neural Computers (Graves et al., 2016) Neural Stacks/Queues (Grefenstette et al., 2015) Learning to Infer Algorithms (Joulin & Mikolov, 2015) Neural Programmer-Interpreters (Reed and de Freitas, 2015) Neural GPUs (Kaiser and Sutskever, 2015)
15 Neural Program Induction (NPI) Data-efficient? Not very Interpretable? No Generalises outside training data? Sometimes Robust to mislabelled data? Yes Robust to ambiguous data? Yes
16 The Best of Both Worlds? SPS NPI Ideally Data-efficient? Yes Not always Yes Interpretable? Yes No Yes Generalises outside training data? Yes Not always Yes Robust to mislabelled data? Not very Yes Yes Robust to ambiguous data? No Yes Yes
17 Neural Program Synthesis (NPS) Given some input/output examples, produce an explicit human-readable program that, when evaluated on the inputs, produces the outputs. Use an optimisation procedure (e.g. gradient descent) to find the program.
18 Neural Program Synthesis (NPS) Given some input/output examples, produce an explicit human-readable program that, when evaluated on the inputs, produces the outputs. Use an optimisation procedure (e.g. gradient descent) to find the program. Examples: ILP, RobustFill, Differentiable Forth, End-to-End Differentiable Proving
19 The Three Approaches Procedure is implicit Symbolic search Optimisation procedure Procedure is explicit Symbolic Program Synthesis Neural Program Induction Neural Program Synthesis
20 The Three Approaches Procedure is implicit Symbolic search Optimisation procedure Procedure is explicit Symbolic Program Synthesis Neural Program Induction Neural Program Synthesis
21 The Three Approaches SPS NPI NPS Data-efficient? Yes Not always Yes Interpretable? Yes No Yes Generalises outside training data? Yes Not always Yes Robust to mislabelled data? No Yes Yes Robust to ambiguous data? No Yes Yes
22 ILP ILP uses a differentiable model of forward chaining inference. The weights represent a probability distribution over clauses. We use SGD to minimise the log-loss. We extract a readable program from the weights.
23 ILP A valuation is a vector in [0,1]ⁿ It maps each of n ground atoms to [0,1]. A valuation represents how likely it is that each of the ground atoms is true.
24 ILP Each clause c is compiled into a function on valuations: For example:
25 ILP We combine the clauses valuations using a weighted sum: We amalgamate the previous valuation with the new clauses valuation: We unroll the network for T steps of forward-chaining inference, generating:
26 ILP ILP uses a differentiable model of forward chaining inference. The weights represent a probability distribution over clauses. We use SGD to minimise the log-loss. We extract a readable program from the weights.
27 ILP Each clause c is compiled into a function on valuations: For example:
28 ILP Assume that each clause has two atoms in the body. For example: We calculate, for each ground atom, the pairs of ground atoms that contribute to its truth:
29 ILP Given our rule: We convert the pairs of atoms into pairs of indices:
30 ILP We convert: into:
31 ILP We convert: into a tensor of shape n * w * 2:
32 ILP We split our tensor X into two matrices of shape n * w : We gather up the results: We take the element-wise product: Here, Z is of shape n * w. Now we take the max across the second dimension:
33
34
35
36
37
38
39
40
41 ILP Experiments
42
43 Example Task: Graph Cyclicity
44 Example Task: Graph Cyclicity cycle(x) pred(x, X). pred(x, Y) edge(x, Y). pred(x, Y) edge(x, Z), pred(z, Y)
45 Example: Fizz-Buzz Fizz 3 Fizz Buzz 15 Fizz+Buzz 6 Fizz Fizz 9 Fizz Buzz 20 Buzz
46 Example: Fizz fizz(x) zero(x). fizz(x) fizz(y), pred1(y, X). pred1(x, Y) succ(x, Z), pred2(z, Y). pred2(x, Y) succ(x, Z), succ(z, Y).
47 Example: Fizz fizz(x) zero(x). fizz(x) fizz(y), pred1(y, X). pred1(x, Y) succ(x, Z), pred2(z, Y). pred2(x, Y) succ(x, Z), succ(z, Y).
48 Example: Buzz buzz(x) zero(x). buzz(x) buzz(y), pred3(y, X). pred3(x, Y) pred1(x, Z), pred2(z, Y). pred1(x, Y) succ(x, Z), pred2(z, Y). pred2(x, Y) succ(x, Z), succ(z, Y).
49 Mis-labelled Data If Symbolic Program Synthesis is given a single mis-labelled piece of training data, it fails catastrophically. We tested ILP with mis-labelled data. We mis-labelled a certain proportion ρ of the training examples. We ran experiments for different values of ρ = 0.0, 0.1, 0.2, 0.3,...
50
51
52 Example: Learning Rules from Ambiguous Data Your system observes: a pair of images a label indicating whether the left image is less than the right image
53 Example: Learning Rules from Ambiguous Data Your system observes: a pair of images a label indicating whether the left image is less than the right image Two forms of generalisation: It must decide if the relation holds for held-out images, and also held-out pairs of digits.
54 Image Generalisation
55 Symbolic Generalisation
56 Symbolic Generalisation NB it has never seen any examples of 2 < 4 in training
57 Symbolic Generalisation 0<1 0<2 0<3 0<4 0<5 0<6 0<7 0<8 0<9 1<2 1<3 1<4 1<5 1<6 1<7 1<8 1<9 2<3 2<4 2<5 2<6 2<7 2<8 2<9 3<4 3<5 3<6 3<7 3<8 3<9 4<5 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 7<9 8<9
58 Symbolic Generalisation 0<1 0<2 0<3 0<4 0<5 0<6 0<7 0<8 0<9 1<2 1<3 1<4 1<5 1<6 1<7 1<8 1<9 2<3 2<4 2<5 2<6 2<7 2<8 2<9 3<4 3<5 3<6 3<7 3<8 3<9 4<5 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 7<9 8<9
59 Symbolic Generalisation 0<1 0<2 0<3 0<4 0<5 1<2 1<3 1<4 1<5 2<3 2<4 2<5 3<4 4<5 0<6 0<7 0<8 0<9 1<7 1<8 1<9 2<6 2<7 2<9 3<6 3<7 3<8 3<9 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 8<9
60 Example: Less Than on MNIST Images Your system observes: a pair of images a label indicating whether the left image is less than the right image Two forms of generalisation: It must decide if the relation holds for held-out images, and also held-out pairs of digits.
61 MLP Baseline We created a baseline MLP to solve this task. The output of the conv-net for the two images is a vector of (20) logits. We added a hidden layer, produced a single output, and trained on cross-entropy loss. The MLP baseline can solve this task easily.
62 Example: Less Than 0<1 0<2 0<3 0<4 0<5 0<6 0<7 0<8 0<9 1<2 1<3 1<4 1<5 1<6 1<7 1<8 1<9 2<3 2<4 2<5 2<6 2<7 2<8 2<9 3<4 3<5 3<6 3<7 3<8 3<9 4<5 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 7<9 8<9
63 Example: Less Than 0<1 0<2 0<3 0<4 0<5 0<6 0<7 0<8 0<9 1<2 1<3 1<4 1<5 1<6 1<7 1<8 1<9 2<3 2<4 2<5 2<6 2<7 2<8 2<9 3<4 3<5 3<6 3<7 3<8 3<9 4<5 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 7<9 8<9
64 Example: Less Than 0<1 0<2 0<3 0<4 0<5 1<2 1<3 1<4 1<5 2<3 2<4 2<5 3<4 4<5 0<6 0<7 0<8 0<9 1<7 1<8 1<9 2<6 2<7 2<9 3<6 3<7 3<8 3<9 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 8<9
65 Example: Less Than 0<1 0<2 0<3 0<4 0<5 1<2 1<3 1<4 1<5 2<3 2<4 2<5 3<4 4<5 0<6 0<7 0<8 0<9 1<7 1<8 1<9 2<6 2<7 2<9 3<6 3<7 3<8 3<9 4<6 4<7 4<8 4<9 5<6 5<7 5<8 5<9 6<7 6<8 6<9 7<8 8<9
66 Example: Less Than 0<1 0<2 0<3 1<2 2<3 0<4 0<5 1<4 1<5 2<4 2<5 3<4 4<5 0<6 0<7 1<7 0<9 1<8 1<9 2<6 2<7 2<9 3<6 3<7 3<8 4<6 4<7 4<8 5<7 5<8 5<9 6<7 6<8 6<9 3<9 7<8 8<9
67 Example: Less Than 0<1 0<2 0<3 1<2 2<3 0<4 0<5 1<4 1<5 2<4 2<5 3<4 4<5 0<6 0<7 1<7 0<9 1<8 1<9 2<6 2<7 2<9 3<6 3<7 3<8 4<6 4<7 4<8 5<7 5<8 5<9 6<7 6<8 6<9 3<9 7<8 8<9
68 Example: Less Than 0<1 0<2 0<4 1<2 1<4 2<3 2<4 0<5 0<6 0<7 1<7 2<5 3<4 4<5 0<9 1<8 1<9 3<9 2<6 2<7 3<6 3<7 3<8 4<6 4<7 4<8 5<7 5<8 5<9 6<9 7<8 8<9
69 Example: Less Than 0<1 0<2 0<4 1<2 1<4 2<3 2<4 0<5 0<6 0<7 1<7 2<5 3<4 4<5 0<9 1<8 1<9 3<9 2<6 2<7 3<6 3<7 3<8 4<6 4<7 4<8 5<7 5<8 5<9 6<9 7<8 8<9
70 Example: Less Than 0<1 0<4 1<2 0<5 0<6 1<4 2<3 2<4 0<7 1<7 2<5 3<4 0<9 1<8 2<7 3<6 4<5 1<9 4<6 3<9 4<7 4<8 5<7 5<8 5<9 6<9 7<8
71 ILP Learning Less-Than We made a slight modification to our original architecture:
72 ILP Learning Less-Than We pre-trained a conv-net to recognise MNIST digits. We convert the logits of the conv-net into a probability distribution over logical atoms. Our model is able to solve this task.
73 ILP Learning Less-Than target() image2(x), pred1(x) pred1(x) image1(y), pred2(y, X) pred2(x, Y) succ(x, Y) pred2(x, Y) pred2(z, Y), pred2(x, Z)
74 Comparing ILP with the Baseline
75 Comparing ILP with the Baseline
76 Conclusion ILP aims to combine the advantages of Symbolic Program Synthesis with the advantages of Neural Program Induction: It is data efficient It can learn interpretable and general rules It is robust to mislabelled data It can handle ambiguous input It can be integrated and trained jointly within larger neural systems/agents
Learning Explanatory Rules from Noisy Data Richard Evans, Ed Grefenstette
Learning Explanatory Rules from Noisy Data Richard Evans, Ed Grefenstette Overview Our system, ILP, learns logic programs from examples. ILP learns by back-propagation. It is robust to noisy and ambiguous
Differentiable Data Structures (and POMDPs)
Differentiable Data Structures (and POMDPs) Yarin Gal & Rowan McAllister February 11, 2016 Many thanks to Edward Grefenstette for graphics material; other sources include Wikimedia licensed under CC BY-SA
Lecture 2 Notes. Outline. Neural Networks. The Big Idea. Architecture. Instructors: Parth Shah, Riju Pahwa
Instructors: Parth Shah, Riju Pahwa Lecture 2 Notes Outline 1. Neural Networks The Big Idea Architecture SGD and Backpropagation 2. Convolutional Neural Networks Intuition Architecture 3. Recurrent Neural
Multi-Layered Perceptrons (MLPs)
Multi-Layered Perceptrons (MLPs) The XOR problem is solvable if we add an extra node to a Perceptron A set of weights can be found for the above 5 connections which will enable the XOR of the inputs to
Machine Learning. MGS Lecture 3: Deep Learning
Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ Machine Learning MGS Lecture 3: Deep Learning Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ WHAT IS DEEP LEARNING? Shallow network: Only one hidden layer
DEEP LEARNING IN PYTHON. The need for optimization
DEEP LEARNING IN PYTHON The need for optimization A baseline neural network Input 2 Hidden Layer 5 2 Output - 9-3 Actual Value of Target: 3 Error: Actual - Predicted = 4 A baseline neural network Input
Global Optimality in Neural Network Training
Global Optimality in Neural Network Training Benjamin D. Haeffele and René Vidal Johns Hopkins University, Center for Imaging Science. Baltimore, USA Questions in Deep Learning Architecture Design Optimization
Lecture 20: Neural Networks for NLP. Zubin Pahuja
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Today s Lecture Feed-forward neural networks as classifiers simple
Cost Functions in Machine Learning
Cost Functions in Machine Learning Kevin Swingler Motivation Given some data that reflects measurements from the environment We want to build a model that reflects certain statistics about that data Something
Deep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
Section 13.2 The Church-Turing Thesis The Church-Turing Thesis: Anything that is intuitively computable can be be computed by a Turing machine.
Section 13.2 The Church-Turing Thesis The Church-Turing Thesis: Anything that is intuitively computable can be be computed by a Turing machine. It is a thesis rather than a theorem because it relates the
Natural Language Processing with Deep Learning CS224N/Ling284. Christopher Manning Lecture 4: Backpropagation and computation graphs
Natural Language Processing with Deep Learning CS4N/Ling84 Christopher Manning Lecture 4: Backpropagation and computation graphs Lecture Plan Lecture 4: Backpropagation and computation graphs 1. Matrix
Recurrent Neural Nets II
Recurrent Neural Nets II Steven Spielberg Pon Kumar, Tingke (Kevin) Shen Machine Learning Reading Group, Fall 2016 9 November, 2016 Outline 1 Introduction 2 Problem Formulations with RNNs 3 LSTM for Optimization
Residual Networks And Attention Models. cs273b Recitation 11/11/2016. Anna Shcherbina
Residual Networks And Attention Models cs273b Recitation 11/11/2016 Anna Shcherbina Introduction to ResNets Introduced in 2015 by Microsoft Research Deep Residual Learning for Image Recognition (He, Zhang,
27: Hybrid Graphical Models and Neural Networks
10-708: Probabilistic Graphical Models 10-708 Spring 2016 27: Hybrid Graphical Models and Neural Networks Lecturer: Matt Gormley Scribes: Jakob Bauer Otilia Stretcu Rohan Varma 1 Motivation We first look
Automatic Differentiation for R
Automatic Differentiation for R Radford M. Neal, University of Toronto Dept. of Statistical Sciences and Dept. of Computer Science Vector Institute Affiliate http://www.cs.utoronto.ca/ radford http://radfordneal.wordpress.com
SGD: Stochastic Gradient Descent
Improving SGD Hantao Zhang Deep Learning with Python Reading: http://neuralnetworksanddeeplearning.com/index.html Chapter 2 SGD: Stochastic Gradient Descent Main Idea: Given a set of input/output examples
Backpropagation + Deep Learning
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Backpropagation + Deep Learning Matt Gormley Lecture 13 Mar 1, 2018 1 Reminders
Variational Autoencoders. Sargur N. Srihari
Variational Autoencoders Sargur N. srihari@cedar.buffalo.edu Topics 1. Generative Model 2. Standard Autoencoder 3. Variational autoencoders (VAE) 2 Generative Model A variational autoencoder (VAE) is a
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision Anonymized for review Abstract Extending the success of deep neural networks to high level tasks like natural language
Motivation. Problem: With our linear methods, we can train the weights but not the basis functions: Activator Trainable weight. Fixed basis function
Neural Networks Motivation Problem: With our linear methods, we can train the weights but not the basis functions: Activator Trainable weight Fixed basis function Flashback: Linear regression Flashback:
Sequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015
Sequence Modeling: Recurrent and Recursive Nets By Pyry Takala 14 Oct 2015 Agenda Why Recurrent neural networks? Anatomy and basic training of an RNN (10.2, 10.2.1) Properties of RNNs (10.2.2, 8.2.6) Using
Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY
Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company
Can Active Memory Replace Attention?
Google Brain NIPS 2016 Presenter: Chao Jiang NIPS 2016 Presenter: Chao Jiang 1 / Outline 1 Introduction 2 Active Memory 3 Step by Step to Neural GPU 4 Another two steps: 1. the Markovian Neural GPU 5 Another
PLT: Inception (cuz there are so many layers)
PLT: Inception (cuz there are so many layers) By: Andrew Aday, (aza2112) Amol Kapoor (ajk2227), Jonathan Zhang (jz2814) Proposal Abstract Overview of domain Purpose Language Outline Types Operators Syntax
Models for grids. Computer vision: models, learning and inference. Multi label Denoising. Binary Denoising. Denoising Goal.
Models for grids Computer vision: models, learning and inference Chapter 9 Graphical Models Consider models where one unknown world state at each pixel in the image takes the form of a grid. Loops in the
Markov Logic: Representation
Markov Logic: Representation Overview Statistical relational learning Markov logic Basic inference Basic learning Statistical Relational Learning Goals: Combine (subsets of) logic and probability into
Usable while performant: the challenges building. Soumith Chintala
Usable while performant: the challenges building Soumith Chintala Problem Statement Deep Learning Workloads Problem Statement Deep Learning Workloads for epoch in range(max_epochs): for data, target in
ImageNet Classification with Deep Convolutional Neural Networks
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky Ilya Sutskever Geoffrey Hinton University of Toronto Canada Paper with same name to appear in NIPS 2012 Main idea Architecture
Improving the way neural networks learn Srikumar Ramalingam School of Computing University of Utah
Improving the way neural networks learn Srikumar Ramalingam School of Computing University of Utah Reference Most of the slides are taken from the third chapter of the online book by Michael Nielson: neuralnetworksanddeeplearning.com
Deep Generative Models Variational Autoencoders
Deep Generative Models Variational Autoencoders Sudeshna Sarkar 5 April 2017 Generative Nets Generative models that represent probability distributions over multiple variables in some way. Directed Generative
Multinomial Regression and the Softmax Activation Function. Gary Cottrell!
Multinomial Regression and the Softmax Activation Function Gary Cottrell Notation reminder We have N data points, or patterns, in the training set, with the pattern number as a superscript: {(x 1,t 1 ),
Deep Learning. Volker Tresp Summer 2014
Deep Learning Volker Tresp Summer 2014 1 Neural Network Winter and Revival While Machine Learning was flourishing, there was a Neural Network winter (late 1990 s until late 2000 s) Around 2010 there
Deep Nets with. Keras
docs https://keras.io Deep Nets with Keras κέρας http://vem.quantumunlimited.org/the-gates-of-horn/ Professor Marie Roch These slides only cover enough to get started with feed-forward networks and do
Tensorflow Example: Fizzbuzz. Sargur N. Srihari
Tensorflow Example: Fizzbuzz Sargur N. srihari@cedar.buffalo.edu 1 Fizzbuzz in Tensorflow Fizzbuzz: Print the numbers from 1 to 100, except that if the number is divisible by 3 print "fizz", if it's divisible
Back Propagation and Other Differentiation Algorithms. Sargur N. Srihari
Back Propagation and Other Differentiation Algorithms Sargur N. srihari@cedar.buffalo.edu 1 Topics (Deep Feedforward Networks) Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units
Dynamic Routing Between Capsules
Report Explainable Machine Learning Dynamic Routing Between Capsules Author: Michael Dorkenwald Supervisor: Dr. Ullrich Köthe 28. Juni 2018 Inhaltsverzeichnis 1 Introduction 2 2 Motivation 2 3 CapusleNet
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius What is Mixed Precision Training? Reduced precision tensor math with FP32 accumulation, FP16 storage Successfully used to train a variety
Deep Learning and Its Applications
Convolutional Neural Network and Its Application in Image Recognition Oct 28, 2016 Outline 1 A Motivating Example 2 The Convolutional Neural Network (CNN) Model 3 Training the CNN Model 4 Issues and Recent
3D Deep Learning on Geometric Forms. Hao Su
3D Deep Learning on Geometric Forms Hao Su Many 3D representations are available Candidates: multi-view images depth map volumetric polygonal mesh point cloud primitive-based CAD models 3D representation
Machine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
CS281 Section 3: Practical Optimization
CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical
Restoring Circuit Structure From SAT Instances. Jarrod A. Roy, Igor L. Markov, and Valeria Bertacco University of Michigan at Ann Arbor
Restoring Circuit Structure From SAT Instances Jarrod A. Roy, Igor L. Markov, and Valeria Bertacco University of Michigan at Ann Arbor Outline Motivation Preliminaries A Generic Circuit Detection Algorithm
Neural Networks. By Laurence Squires
Neural Networks By Laurence Squires Machine learning What is it? Type of A.I. (possibly the ultimate A.I.?!?!?!) Algorithms that learn how to classify data The algorithms slowly change their own variables
Department of Electrical and Computer Engineering
LAGRANGIAN RELAXATION FOR GATE IMPLEMENTATION SELECTION Yi-Le Huang, Jiang Hu and Weiping Shi Department of Electrical and Computer Engineering Texas A&M University OUTLINE Introduction and motivation
Deep Learning with R. Francesca Lazzeri Data Scientist II - Microsoft, AI Research
with R Francesca Lazzeri - @frlazzeri Data Scientist II - Microsoft, AI Research Agenda with R What is Demo Better understanding of R DL tools Fundamental concepts in Forward Propagation Algorithm Activation
SEMANTIC COMPUTING. Lecture 8: Introduction to Deep Learning. TU Dresden, 7 December Dagmar Gromann International Center For Computational Logic
SEMANTIC COMPUTING Lecture 8: Introduction to Deep Learning Dagmar Gromann International Center For Computational Logic TU Dresden, 7 December 2018 Overview Introduction Deep Learning General Neural Networks
Unsupervised Learning
Deep Learning for Graphics Unsupervised Learning Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Kostas Rematas Tobias Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL Timetable Niloy
Keras: Handwritten Digit Recognition using MNIST Dataset
Keras: Handwritten Digit Recognition using MNIST Dataset IIT PATNA February 9, 2017 1 / 24 OUTLINE 1 Introduction Keras: Deep Learning library for Theano and TensorFlow 2 Installing Keras Installation
Recurrent Neural Networks. Nand Kishore, Audrey Huang, Rohan Batra
Recurrent Neural Networks Nand Kishore, Audrey Huang, Rohan Batra Roadmap Issues Motivation 1 Application 1: Sequence Level Training 2 Basic Structure 3 4 Variations 5 Application 3: Image Classification
Deconvolution Networks
Deconvolution Networks Johan Brynolfsson Mathematical Statistics Centre for Mathematical Sciences Lund University December 6th 2016 1 / 27 Deconvolution Neural Networks 2 / 27 Image Deconvolution True
A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
A Neural Algorithm of Artistic Style Leon A. Gatys, Alexander S. Ecker, Matthias Bethge Presented by Shishir Mathur (1 Sept 2016) What is this paper This is the research paper behind Prisma It creates
Geometric Reconstruction Dense reconstruction of scene geometry
Lecture 5. Dense Reconstruction and Tracking with Real-Time Applications Part 2: Geometric Reconstruction Dr Richard Newcombe and Dr Steven Lovegrove Slide content developed from: [Newcombe, Dense Visual
Perceptron: This is convolution!
Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image
Functional Programming in Embedded System Design
Functional Programming in Embedded System Design Al Strelzoff CTO, E-TrolZ Lawrence, Mass. al.strelzoff@e-trolz.com 1/19/2006 Copyright E-TrolZ, Inc. 1 of 20 Hard Real Time Embedded Systems MUST: Be highly
Learning. Learning agents Inductive learning. Neural Networks. Different Learning Scenarios Evaluation
Learning Learning agents Inductive learning Different Learning Scenarios Evaluation Slides based on Slides by Russell/Norvig, Ronald Williams, and Torsten Reil Material from Russell & Norvig, chapters
Logical Rhythm - Class 3. August 27, 2018
Logical Rhythm - Class 3 August 27, 2018 In this Class Neural Networks (Intro To Deep Learning) Decision Trees Ensemble Methods(Random Forest) Hyperparameter Optimisation and Bias Variance Tradeoff Biological
Optimization in the Big Data Regime 5: Parallelization? Sham M. Kakade
Optimization in the Big Data Regime 5: Parallelization? Sham M. Kakade Machine Learning for Big Data CSE547/STAT548 University of Washington S. M. Kakade (UW) Optimization for Big data 1 / 21 Announcements...
CS 224N: Assignment #1
Due date: assignment) 1/25 11:59 PM PST (You are allowed to use three (3) late days maximum for this These questions require thought, but do not require long answers. Please be as concise as possible.
Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic
Philippe Giguère Overall Description Goal: to improve spatial invariance to the input data Translation, Rotation, Scale, Clutter, Elastic How: add a learnable module which explicitly manipulate spatially
Lecture 37: ConvNets (Cont d) and Training
Lecture 37: ConvNets (Cont d) and Training CS 4670/5670 Sean Bell [http://bbabenko.tumblr.com/post/83319141207/convolutional-learnings-things-i-learned-by] (Unrelated) Dog vs Food [Karen Zack, @teenybiscuit]
Evaluation. Evaluate what? For really large amounts of data... A: Use a validation set.
Evaluate what? Evaluation Charles Sutton Data Mining and Exploration Spring 2012 Do you want to evaluate a classifier or a learning algorithm? Do you want to predict accuracy or predict which one is better?
Deep Model Compression
Deep Model Compression Xin Wang Oct.31.2016 Some of the contents are borrowed from Hinton s and Song s slides. Two papers Distilling the Knowledge in a Neural Network by Geoffrey Hinton et al What s the
DEEP LEARNING REVIEW. Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature Presented by Divya Chitimalla
DEEP LEARNING REVIEW Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature 2015 -Presented by Divya Chitimalla What is deep learning Deep learning allows computational models that are composed of multiple
BHNN: a Memory-Efficient Accelerator for Compressing Deep Neural Network with Blocked Hashing Techniques
BHNN: a Memory-Efficient Accelerator for Compressing Deep Neural Network with Blocked Hashing Techniques Jingyang Zhu 1, Zhiliang Qian 2*, and Chi-Ying Tsui 1 1 The Hong Kong University of Science and
Manifold Constrained Deep Neural Networks for ASR
1 Manifold Constrained Deep Neural Networks for ASR Department of Electrical and Computer Engineering, McGill University Richard Rose and Vikrant Tomar Motivation Speech features can be characterized as
Natural Language Processing CS 6320 Lecture 6 Neural Language Models. Instructor: Sanda Harabagiu
Natural Language Processing CS 6320 Lecture 6 Neural Language Models Instructor: Sanda Harabagiu In this lecture We shall cover: Deep Neural Models for Natural Language Processing Introduce Feed Forward
Final Report: Classification of Plankton Classes By Tae Ho Kim and Saaid Haseeb Arshad
Final Report: Classification of Plankton Classes By Tae Ho Kim and Saaid Haseeb Arshad Table of Contents 1. Project Overview a. Problem Statement b. Data c. Overview of the Two Stages of Implementation
Pattern Recognition. Kjell Elenius. Speech, Music and Hearing KTH. March 29, 2007 Speech recognition
Pattern Recognition Kjell Elenius Speech, Music and Hearing KTH March 29, 2007 Speech recognition 2007 1 Ch 4. Pattern Recognition 1(3) Bayes Decision Theory Minimum-Error-Rate Decision Rules Discriminant
Neural Nets for Adaptive Filter and Adaptive Pattern Recognition
Adaptive Pattern btyoung@gmail.com CSCE 636 10 February 2010 Outline Adaptive Combiners and Filters Minimal Disturbance and the Algorithm Madaline Rule II () Published 1988 in IEEE Journals Bernard Widrow
Akarsh Pokkunuru EECS Department Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Akarsh Pokkunuru EECS Department 03-16-2017 Contractive Auto-Encoders: Explicit Invariance During Feature Extraction 1 AGENDA Introduction to Auto-encoders Types of Auto-encoders Analysis of different
Neural Programming by Example
Neural Programming by Example Chengxun Shu Beihang University Beijing 100191, China shuchengxun@163.com Hongyu Zhang The University of Newcastle Callaghan, NSW 2308, Australia hongyu.zhang@newcastle.edu.au
Inductive Logic Programming in Clementine
Inductive Logic Programming in Clementine Sam Brewer 1 and Tom Khabaza 2 Advanced Data Mining Group, SPSS (UK) Ltd 1st Floor, St. Andrew s House, West Street Woking, Surrey GU21 1EB, UK 1 sbrewer@spss.com,
register allocation saves energy register allocation reduces memory accesses.
Lesson 10 Register Allocation Full Compiler Structure Embedded systems need highly optimized code. This part of the course will focus on Back end code generation. Back end: generation of assembly instructions
Reddit Recommendation System Daniel Poon, Yu Wu, David (Qifan) Zhang CS229, Stanford University December 11 th, 2011
Reddit Recommendation System Daniel Poon, Yu Wu, David (Qifan) Zhang CS229, Stanford University December 11 th, 2011 1. Introduction Reddit is one of the most popular online social news websites with millions
Asynchronous Parallel Stochastic Gradient Descent. A Numeric Core for Scalable Distributed Machine Learning Algorithms
Asynchronous Parallel Stochastic Gradient Descent A Numeric Core for Scalable Distributed Machine Learning Algorithms J. Keuper and F.-J. Pfreundt Competence Center High Performance Computing Fraunhofer
Score function estimator and variance reduction techniques
and variance reduction techniques Wilker Aziz University of Amsterdam May 24, 2018 Wilker Aziz Discrete variables 1 Outline 1 2 3 Wilker Aziz Discrete variables 1 Variational inference for belief networks
DECISION TREES & RANDOM FORESTS X CONVOLUTIONAL NEURAL NETWORKS
DECISION TREES & RANDOM FORESTS X CONVOLUTIONAL NEURAL NETWORKS Deep Neural Decision Forests Microsoft Research Cambridge UK, ICCV 2015 Decision Forests, Convolutional Networks and the Models in-between
Lecture 25: Bezier Subdivision. And he took unto him all these, and divided them in the midst, and laid each piece one against another: Genesis 15:10
Lecture 25: Bezier Subdivision And he took unto him all these, and divided them in the midst, and laid each piece one against another: Genesis 15:10 1. Divide and Conquer If we are going to build useful
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN) By Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Convolution on Image I. 1.1. Convolution in 1D II. 1.2. Convolution
Uncertain Data Models
Uncertain Data Models Christoph Koch EPFL Dan Olteanu University of Oxford SYNOMYMS data models for incomplete information, probabilistic data models, representation systems DEFINITION An uncertain data
Assignment # 5. Farrukh Jabeen Due Date: November 2, Neural Networks: Backpropation
Farrukh Jabeen Due Date: November 2, 2009. Neural Networks: Backpropation Assignment # 5 The "Backpropagation" method is one of the most popular methods of "learning" by a neural network. Read the class
Knowledge Discovery and Data Mining. Neural Nets. A simple NN as a Mathematical Formula. Notes. Lecture 13 - Neural Nets. Tom Kelsey.
Knowledge Discovery and Data Mining Lecture 13 - Neural Nets Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-13-NN
Decentralized and Distributed Machine Learning Model Training with Actors
Decentralized and Distributed Machine Learning Model Training with Actors Travis Addair Stanford University taddair@stanford.edu Abstract Training a machine learning model with terabytes to petabytes of
Stacked Denoising Autoencoders for Face Pose Normalization
Stacked Denoising Autoencoders for Face Pose Normalization Yoonseop Kang 1, Kang-Tae Lee 2,JihyunEun 2, Sung Eun Park 2 and Seungjin Choi 1 1 Department of Computer Science and Engineering Pohang University
Semantics. A. Demers Jan This material is primarily from Ch. 2 of the text. We present an imperative
CS411 Notes 1: IMP and Large Step Operational Semantics A. Demers 23-25 Jan 2001 This material is primarily from Ch. 2 of the text. We present an imperative language called IMP; wegive a formal definition
Knowledge Discovery and Data Mining
Knowledge Discovery and Data Mining Lecture 13 - Neural Nets Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-13-NN
Neuron Selectivity as a Biologically Plausible Alternative to Backpropagation
Neuron Selectivity as a Biologically Plausible Alternative to Backpropagation C.J. Norsigian Department of Bioengineering cnorsigi@eng.ucsd.edu Vishwajith Ramesh Department of Bioengineering vramesh@eng.ucsd.edu
Generative Adversarial Networks (GANs) Ian Goodfellow, Research Scientist MLSLP Keynote, San Francisco
Generative Adversarial Networks (GANs) Ian Goodfellow, Research Scientist MLSLP Keynote, San Francisco 2016-09-13 Generative Modeling Density estimation Sample generation Training examples Model samples
Model learning for robot control: a survey
Model learning for robot control: a survey Duy Nguyen-Tuong, Jan Peters 2011 Presented by Evan Beachly 1 Motivation Robots that can learn how their motors move their body Complexity Unanticipated Environments
291 Programming Assignment #3
000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050
Automated Crystal Structure Identification from X-ray Diffraction Patterns
Automated Crystal Structure Identification from X-ray Diffraction Patterns Rohit Prasanna (rohitpr) and Luca Bertoluzzi (bertoluz) CS229: Final Report 1 Introduction X-ray diffraction is a commonly used
The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.
CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or
BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks Surat Teerapittayanon Harvard University Email: steerapi@seas.harvard.edu Bradley McDanel Harvard University Email: mcdanel@fas.harvard.edu
arxiv: v1 [cond-mat.dis-nn] 30 Dec 2018
![arxiv: v1 [cond-mat.dis-nn] 30 Dec 2018](/thumbs/93/111774960.jpg "arxiv: v1 [cond-mat.dis-nn] 30 Dec 2018")
A General Deep Learning Framework for Structure and Dynamics Reconstruction from Time Series Data arxiv:1812.11482v1 [cond-mat.dis-nn] 30 Dec 2018 Zhang Zhang, Jing Liu, Shuo Wang, Ruyue Xin, Jiang Zhang
Clustering with gradient descent
Clustering with gradient descent D. Derkach 34 N. Kazeev 3 R. Neychev 23 A. Panin 3 F. Ratnikov 34 I. Trofimov 1 M. Vesterinen 5 A. Ustyuzhanin 234 1 Yandex Data Factory 2 MIPT 3 Yandex School of Data
Hidden Units. Sargur N. Srihari
Hidden Units Sargur N. srihari@cedar.buffalo.edu 1 Topics in Deep Feedforward Networks Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation
Interaction Nets vs. the ρ-calculus: Introducing Bigraphical Nets
Interaction Nets vs. the ρ-calculus: Introducing Bigraphical Nets Maribel Fernández 1 Ian Mackie 1 François-Régis Sinot 2 1 DCS, King s College London 2 LIX, École Polytechnique Rho-Calculus Workshop,
An introduction to neural networks for beginners. By Dr Andy Thomas Adventures in Machine Learning
An introduction to neural networks for beginners By Dr Andy Thomas Adventures in Machine Learning Table of Contents Introduction... 2 Who I am and my approach... 2 The code, pre-requisites and installation...
Declarative programming. Logic programming is a declarative style of programming.
Declarative programming Logic programming is a declarative style of programming. Declarative programming Logic programming is a declarative style of programming. The programmer says what they want to compute,