# Neural Network Neurons

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Neural Networks

2 Neural Network Neurons 1 Receives n inputs (plus a bias term) Multiplies each input by its weight Applies activation function to the sum of results Outputs result

3 Activation Functions Given the weighted sum as input, an activation function controls whether a neuron is active or inactive Commonly used activation functions: o Threshold function: outputs 1 when input is positive and 0 otherwise (similar to perceptron) o Sigmoid function: Differentiable a good property for optimization

4 Basic Multilayer Neural Network Output layer Hidden layer x 1 x 2 x 3 x 4 Input layer Each layer receives its inputs from the previous layer and forwards its outputs to the next feed forward structure Output layer: sigmoid activation function for classification, and linear activation function for regression Referred to as a two-layer network (2 layer of weights)

5 Representational Power Any Boolean Formula Consider a formula in disjunctive normal form: OR units AND units x 1 x 2 x 3

6

7

8 Training: Backpropagation It is traditional to train neural networks to minimize the squared-error This is appropriate for regression problems, but for classification problems it is more appropriate to optimize log-likelihood But we will look at MSE as our example Gradient descent: must apply the chain rule many times to compute the gradient

9 Terminology a 9 W 9 a 6 a 7 a 8 W 6 W 7 W 8 x 1 x 2 x 3 x 4 1, 1, 2, 3, 4 the input vector with the bias term 1, 6, 7, 8 the output of the hidden layer with the bias term represents the weight vector leading to node, represents the weight connecting from the -th node to the -th node 9, 6 is the weight connecting from 6 to 9 We will use to represent the activation function, so

10 Mean Squared Error We adjust the weights of the neural network to minimize the mean squared error (MSE) on training set. Useful fact: the derivative of the sigmoid activation function is 1

11 Gradient Descent: Output Unit a 9 W 9 a 6 a 7 a 8 W 6 W 7 W 8 w 9,6 is a component of W 9, connecting from a 6 to the output node. x 1 x 2 x 3 x 4 1

12 The Delta Rule a 9 W 9 a 6 a 7 a 8 W 6 W 7 W 8 w 9,6 x 1 x 2 x 3 x 4 Define then

13 Derivation: Hidden Units a 9 W 9 a 6 a 7 a 8 W 6 W 7 W 8 w 6,2 x 1 x 2 x 3 x 4

14 Delta Rule for Hidden Units a 9 W 9 a 6 a 7 a 8 W 6 W 7 W 8 Define and rewrite as w 6,2 x 1 x 2 x 3 x 4

15 Networks with Multiple Output Units a 9 a 10 W 9 a 6 a 7 a 8 W 6 W 7 W 8 x 1 x 2 x 3 x 4 We get a separate contribution to the gradient from each output unit. Hence, for input-to-hidden weights, we must sum up the contributions:

16 Backpropagation Training Initialize all the weights with small random values Repeat For all training examples, do Begin Epoch For each training example do Compute the network output Compute the error Backpropagate this error from layer to layer and adjust weights to decrease this error End Epoch

17 Backpropagation for a Single Example Forward Pass Given, compute a u and ŷ v for hidden units u and output units v. Compute Errors Compute ŷ for each output unit v Compute Output Deltas ŷ ŷ 1 ŷ Compute Hidden Deltas 1 Σ, Compute Gradient Compute for hidden-to-output weights. Compute for input-to-hidden weights. Backpropagate error from layer to layer Take Gradient Descent Step,,,,,,

18 Remarks on Training No guarantee of convergence, may oscillate or reach a local minima. However, in practice many large networks can be adequately trained on large amounts of data for realistic problems, e.g., Driving a car Recognizing handwritten zip codes Play world championship level Backgammon Many epochs (thousands) may be needed for adequate training, large data sets may require hours or days of CPU time. Termination criteria can be: Fixed number of epochs Threshold on training set error Increased error on a validation set To avoid local minima problems, can run several trials starting from different initial random weights and: Take the result with the best training or validation performance. Build a committee of networks that vote during testing, possibly weighting vote by training or validation accuracy

19 Notes on Proper Initialization Start in the linear regions keep all weights near zero, so that all sigmoid units are in their linear regions. This makes the whole net the equivalent of one linear threshold unit a relatively simple function. This will also avoid having very small gradient Break symmetry If we start with all the weights equal, what would happen? Ensure that each hidden unit has different input weights so that the hidden units move in different directions. Set each weight to a random number in the range where the fan-in of weight w v,u is the number of inputs to unit v.

20

21 Overtraining Prevention Running too many epochs may overtrain the network and result in overfitting. Keep a validation set and test accuracy after every epoch. Maintain weights for best performing network on the validation set and return it when performance decreases significantly beyond this. To avoid losing training data to validation: Use 10-fold cross-validation to determine the average number of epochs that optimizes validation performance. Train on the full data set using this many epochs to produce the final result.

22 Over-fitting Prevention Too few hidden units prevent the system from adequately fitting the data and learning the concept. Too many hidden units leads to over-fitting. Similar cross-validation method can be used to decide an appropriate number of hidden units. Another approach to preventing over-fitting is weight decay, in which we multiply all weights by some fraction between 0 and 1 after each epoch. - Encourages smaller weights and less complex hypotheses. - Equivalent to including an additive penalty in the error function proportional to the sum of the squares of the weights of the network.

23 Input/Output Coding Appropriate coding of inputs/outputs can make learning easier and improve generalization. Best to encode discrete multi-category features using multiple input units and include one binary unit per value Continuous inputs can be handled by a single input unit, but scaling them between 0 and 1 For classification problems, best to have one output unit per class. Continuous output values then represent certainty in various classes. Assign test instances to the class with the highest output. Use target values of 0.9 and 0.1 for binary problems rather than forcing weights to grow large enough to closely approximate 0/1 outputs. Continuous outputs (regression) can also be handled by scaling to the range between 0 and 1

24 Softmax for multi-class classification For K classes, we have K nodes in the output layer, one for each class Let be the output of the class- node, i.e., where is the output of the hidden layer, and is the weight vector leading into the class-k node We define: ŷ 1 ŷ 2 softmax W 9 W 10 1 a 6 a 7 a 8 W 6 W 7 W 8 1 x 1 x 2 x 3 x 4

25 Likelihood objective Given an instance, we compute the outputs for : These will be the probabilities computed using the softmax, and they will be a function of the weights of the neural net (thus the subscript ) Log-likelihood function: Similar gradient descent algorithm could be derived

26 Recent Development A recent trend in ML is deep learning, which learns feature hierarchies from large amounts of unlabeled data The feature hierarchies are expected to capture the inherent structure in the data Can often lead to better classification when used the learned features to train with labeled data Neural networks provide one approach for deep learning

27 Shallow vs Deep Architectures Traditional shallow architecture Image Video Text Class label Image Video Text Deep architecture Class label Learned feature representation

28 Deep learning with Auto-encoders Network is trained to output the input The hidden layer serves to extract features from the input that is essential for representation Constraining layer 2 to be sparse Training is carried out with the same back-prop procedure

29 Deep structure: Stacked Auto-encoders This continues and learns a feature hierarchy The final feature layer can then be used as features for supervised learning Can also do supervised training on the entire network to finetune all weights

30 That s the basic idea There are many many types of deep learning Different kinds of autoencoder, variations on architectures and training algorithms, etc Various packages: Theano, Caffe, Torch, Convnet Tremendous impact in vision, speech and natural language processing Very fast growing area To learn more about deep learning, take Dr. Fuxin Li s CS519 next term

### Neural Network Learning. Today s Lecture. Continuation of Neural Networks. Artificial Neural Networks. Lecture 24: Learning 3. Victor R.

Lecture 24: Learning 3 Victor R. Lesser CMPSCI 683 Fall 2010 Today s Lecture Continuation of Neural Networks Artificial Neural Networks Compose of nodes/units connected by links Each link has a numeric

### Supervised Learning in Neural Networks (Part 2)

Supervised Learning in Neural Networks (Part 2) Multilayer neural networks (back-propagation training algorithm) The input signals are propagated in a forward direction on a layer-bylayer basis. Learning

### Perceptrons and Backpropagation. Fabio Zachert Cognitive Modelling WiSe 2014/15

Perceptrons and Backpropagation Fabio Zachert Cognitive Modelling WiSe 2014/15 Content History Mathematical View of Perceptrons Network Structures Gradient Descent Backpropagation (Single-Layer-, Multilayer-Networks)

### CS 1674: Intro to Computer Vision. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh November 16, 2016

CS 1674: Intro to Computer Vision Neural Networks Prof. Adriana Kovashka University of Pittsburgh November 16, 2016 Announcements Please watch the videos I sent you, if you haven t yet (that s your reading)

### Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group

Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group 1D Input, 1D Output target input 2 2D Input, 1D Output: Data Distribution Complexity Imagine many dimensions (data occupies

### LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS

LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS Neural Networks Classifier Introduction INPUT: classification data, i.e. it contains an classification (class) attribute. WE also say that the class

### Introduction to Deep Learning

ENEE698A : Machine Learning Seminar Introduction to Deep Learning Raviteja Vemulapalli Image credit: [LeCun 1998] Resources Unsupervised feature learning and deep learning (UFLDL) tutorial (http://ufldl.stanford.edu/wiki/index.php/ufldl_tutorial)

### Learning. Learning agents Inductive learning. Neural Networks. Different Learning Scenarios Evaluation

Learning Learning agents Inductive learning Different Learning Scenarios Evaluation Slides based on Slides by Russell/Norvig, Ronald Williams, and Torsten Reil Material from Russell & Norvig, chapters

### Review on Methods of Selecting Number of Hidden Nodes in Artificial Neural Network

Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 3, Issue. 11, November 2014,

### CP365 Artificial Intelligence

CP365 Artificial Intelligence Tech News! Apple news conference tomorrow? Tech News! Apple news conference tomorrow? Google cancels Project Ara modular phone Weather-Based Stock Market Predictions? Dataset

### Perceptron: This is convolution!

Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image

### Using neural nets to recognize hand-written digits. Srikumar Ramalingam School of Computing University of Utah

Using neural nets to recognize hand-written digits Srikumar Ramalingam School of Computing University of Utah Reference Most of the slides are taken from the first chapter of the online book by Michael

### Machine Learning Classifiers and Boosting

Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Naïve

### Optimizing Number of Hidden Nodes for Artificial Neural Network using Competitive Learning Approach

Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 5, May 2015, pg.358

### A Systematic Overview of Data Mining Algorithms

A Systematic Overview of Data Mining Algorithms 1 Data Mining Algorithm A well-defined procedure that takes data as input and produces output as models or patterns well-defined: precisely encoded as a

### COMPUTATIONAL INTELLIGENCE

COMPUTATIONAL INTELLIGENCE Fundamentals Adrian Horzyk Preface Before we can proceed to discuss specific complex methods we have to introduce basic concepts, principles, and models of computational intelligence

### Deep Learning. Deep Learning provided breakthrough results in speech recognition and image classification. Why?

Data Mining Deep Learning Deep Learning provided breakthrough results in speech recognition and image classification. Why? Because Speech recognition and image classification are two basic examples of

### Deep Learning. Volker Tresp Summer 2014

Deep Learning Volker Tresp Summer 2014 1 Neural Network Winter and Revival While Machine Learning was flourishing, there was a Neural Network winter (late 1990 s until late 2000 s) Around 2010 there

### Multinomial Regression and the Softmax Activation Function. Gary Cottrell!

Multinomial Regression and the Softmax Activation Function Gary Cottrell Notation reminder We have N data points, or patterns, in the training set, with the pattern number as a superscript: {(x 1,t 1 ),

### Neural Network Weight Selection Using Genetic Algorithms

Neural Network Weight Selection Using Genetic Algorithms David Montana presented by: Carl Fink, Hongyi Chen, Jack Cheng, Xinglong Li, Bruce Lin, Chongjie Zhang April 12, 2005 1 Neural Networks Neural networks

### COMPUTATIONAL INTELLIGENCE

COMPUTATIONAL INTELLIGENCE Radial Basis Function Networks Adrian Horzyk Preface Radial Basis Function Networks (RBFN) are a kind of artificial neural networks that use radial basis functions (RBF) as activation

### A Systematic Overview of Data Mining Algorithms. Sargur Srihari University at Buffalo The State University of New York

A Systematic Overview of Data Mining Algorithms Sargur Srihari University at Buffalo The State University of New York 1 Topics Data Mining Algorithm Definition Example of CART Classification Iris, Wine

### Deep Learning for Computer Vision II

IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L

### Deep Learning Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD.

Deep Learning 861.061 Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD asan.agibetov@meduniwien.ac.at Medical University of Vienna Center for Medical Statistics,

### Why MultiLayer Perceptron/Neural Network? Objective: Attributes:

Why MultiLayer Perceptron/Neural Network? Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are

### MLPQNA-LEMON Multi Layer Perceptron neural network trained by Quasi Newton or Levenberg-Marquardt optimization algorithms

MLPQNA-LEMON Multi Layer Perceptron neural network trained by Quasi Newton or Levenberg-Marquardt optimization algorithms 1 Introduction In supervised Machine Learning (ML) we have a set of data points

### Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal

Object Detection Lecture 10.3 - Introduction to deep learning (CNN) Idar Dyrdal Deep Learning Labels Computational models composed of multiple processing layers (non-linear transformations) Used to learn

### Neural Networks Laboratory EE 329 A

Neural Networks Laboratory EE 329 A Introduction: Artificial Neural Networks (ANN) are widely used to approximate complex systems that are difficult to model using conventional modeling techniques such

### Neural Networks (Overview) Prof. Richard Zanibbi

Neural Networks (Overview) Prof. Richard Zanibbi Inspired by Biology Introduction But as used in pattern recognition research, have little relation with real neural systems (studied in neurology and neuroscience)

### Clustering algorithms and autoencoders for anomaly detection

Clustering algorithms and autoencoders for anomaly detection Alessia Saggio Lunch Seminars and Journal Clubs Université catholique de Louvain, Belgium 3rd March 2017 a Outline Introduction Clustering algorithms

### Univariate and Multivariate Decision Trees

Univariate and Multivariate Decision Trees Olcay Taner Yıldız and Ethem Alpaydın Department of Computer Engineering Boğaziçi University İstanbul 80815 Turkey Abstract. Univariate decision trees at each

### β-release Multi Layer Perceptron Trained by Quasi Newton Rule MLPQNA User Manual

β-release Multi Layer Perceptron Trained by Quasi Newton Rule MLPQNA User Manual DAME-MAN-NA-0015 Issue: 1.0 Date: July 28, 2011 Author: M. Brescia, S. Riccardi Doc. : BetaRelease_Model_MLPQNA_UserManual_DAME-MAN-NA-0015-Rel1.0

### Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech

Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:

### 6. Backpropagation training 6.1 Background

6. Backpropagation training 6.1 Background To understand well how a feedforward neural network is built and it functions, we consider its basic first steps. We return to its history for a while. In 1949

### ImageNet Classification with Deep Convolutional Neural Networks

ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky Ilya Sutskever Geoffrey Hinton University of Toronto Canada Paper with same name to appear in NIPS 2012 Main idea Architecture

### Fast Learning for Big Data Using Dynamic Function

IOP Conference Series: Materials Science and Engineering PAPER OPEN ACCESS Fast Learning for Big Data Using Dynamic Function To cite this article: T Alwajeeh et al 2017 IOP Conf. Ser.: Mater. Sci. Eng.

### Neural Networks. Robot Image Credit: Viktoriya Sukhanova 123RF.com

Neural Networks These slides were assembled by Eric Eaton, with grateful acknowledgement of the many others who made their course materials freely available online. Feel free to reuse or adapt these slides

### Neural Networks. Robot Image Credit: Viktoriya Sukhanova 123RF.com

Neural Networks These slides were assembled by Eric Eaton, with grateful acknowledgement of the many others who made their course materials freely available online. Feel free to reuse or adapt these slides

### In this assignment, we investigated the use of neural networks for supervised classification

Paul Couchman Fabien Imbault Ronan Tigreat Gorka Urchegui Tellechea Classification assignment (group 6) Image processing MSc Embedded Systems March 2003 Classification includes a broad range of decision-theoric

### Lecture #11: The Perceptron

Lecture #11: The Perceptron Mat Kallada STAT2450 - Introduction to Data Mining Outline for Today Welcome back! Assignment 3 The Perceptron Learning Method Perceptron Learning Rule Assignment 3 Will be

### Sequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015

Sequence Modeling: Recurrent and Recursive Nets By Pyry Takala 14 Oct 2015 Agenda Why Recurrent neural networks? Anatomy and basic training of an RNN (10.2, 10.2.1) Properties of RNNs (10.2.2, 8.2.6) Using

### 6. Convolutional Neural Networks

6. Convolutional Neural Networks CS 519 Deep Learning, Winter 2017 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Thursday (2/2) 20 minutes Topics: Optimization Basic neural networks No Convolutional

### Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center

Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction

### What is machine learning?

Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship

### Neural Bag-of-Features Learning

Neural Bag-of-Features Learning Nikolaos Passalis, Anastasios Tefas Department of Informatics, Aristotle University of Thessaloniki Thessaloniki 54124, Greece Tel,Fax: +30-2310996304 Abstract In this paper,

### Traffic Signs Recognition using HP and HOG Descriptors Combined to MLP and SVM Classifiers

Traffic Signs Recognition using HP and HOG Descriptors Combined to MLP and SVM Classifiers A. Salhi, B. Minaoui, M. Fakir, H. Chakib, H. Grimech Faculty of science and Technology Sultan Moulay Slimane

### Como funciona o Deep Learning

Como funciona o Deep Learning Moacir Ponti (com ajuda de Gabriel Paranhos da Costa) ICMC, Universidade de São Paulo Contact: www.icmc.usp.br/~moacir moacir@icmc.usp.br Uberlandia-MG/Brazil October, 2017

### Scaling Distributed Machine Learning

Scaling Distributed Machine Learning with System and Algorithm Co-design Mu Li Thesis Defense CSD, CMU Feb 2nd, 2017 nx min w f i (w) Distributed systems i=1 Large scale optimization methods Large-scale

### Assignment 2. Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions

ENEE 739Q: STATISTICAL AND NEURAL PATTERN RECOGNITION Spring 2002 Assignment 2 Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions Aravind Sundaresan

### Machine Learning Lecture-1

Machine Learning Lecture-1 Programming Club Indian Institute of Technology Kanpur pclubiitk@gmail.com February 25, 2016 Programming Club (IIT Kanpur) ML-1 February 25, 2016 1 / 18 Acknowledgement This

### Artificial Neural Networks. Introduction to Computational Neuroscience Ardi Tampuu

Artificial Neural Networks Introduction to Computational Neuroscience Ardi Tampuu 7.0.206 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition

### Understanding Andrew Ng s Machine Learning Course Notes and codes (Matlab version)

Understanding Andrew Ng s Machine Learning Course Notes and codes (Matlab version) Note: All source materials and diagrams are taken from the Coursera s lectures created by Dr Andrew Ng. Everything I have

### CHAPTER VI BACK PROPAGATION ALGORITHM

6.1 Introduction CHAPTER VI BACK PROPAGATION ALGORITHM In the previous chapter, we analysed that multiple layer perceptrons are effectively applied to handle tricky problems if trained with a vastly accepted

### Adversarial Examples and Adversarial Training. Ian Goodfellow, Staff Research Scientist, Google Brain CS 231n, Stanford University,

Adversarial Examples and Adversarial Training Ian Goodfellow, Staff Research Scientist, Google Brain CS 231n, Stanford University, 2017-05-30 Overview What are adversarial examples? Why do they happen?

### Cost Functions in Machine Learning

Cost Functions in Machine Learning Kevin Swingler Motivation Given some data that reflects measurements from the environment We want to build a model that reflects certain statistics about that data Something

### Backpropagation and Neural Networks. Lecture 4-1

Lecture 4: Backpropagation and Neural Networks Lecture 4-1 Administrative Assignment 1 due Thursday April 20, 11:59pm on Canvas Lecture 4-2 Administrative Project: TA specialities and some project ideas

### Practical Methodology. Lecture slides for Chapter 11 of Deep Learning Ian Goodfellow

Practical Methodology Lecture slides for Chapter 11 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-26 What drives success in ML? Arcane knowledge of dozens of obscure algorithms? Mountains

### What is Learning? CS 343: Artificial Intelligence Machine Learning. Raymond J. Mooney. Problem Solving / Planning / Control.

What is Learning? CS 343: Artificial Intelligence Machine Learning Herbert Simon: Learning is any process by which a system improves performance from experience. What is the task? Classification Problem

### CSE 5526: Introduction to Neural Networks Radial Basis Function (RBF) Networks

CSE 5526: Introduction to Neural Networks Radial Basis Function (RBF) Networks Part IV 1 Function approximation MLP is both a pattern classifier and a function approximator As a function approximator,

### Linear Separability. Linear Separability. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons

Linear Separability Input space in the two-dimensional case (n = ): - - - - - - w =, w =, = - - - - - - w = -, w =, = - - - - - - w = -, w =, = Linear Separability So by varying the weights and the threshold,

### CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS

CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of

### Recitation Supplement: Creating a Neural Network for Classification SAS EM December 2, 2002

Recitation Supplement: Creating a Neural Network for Classification SAS EM December 2, 2002 Introduction Neural networks are flexible nonlinear models that can be used for regression and classification

### Subgoal Chaining and the Local Minimum Problem

Subgoal Chaining and the Local Minimum roblem Jonathan. Lewis (jonl@dcs.st-and.ac.uk), Michael K. Weir (mkw@dcs.st-and.ac.uk) Department of Computer Science, University of St. Andrews, St. Andrews, Fife

### Distributed Optimization of Deeply Nested Systems

Distributed Optimization of Deeply Nested Systems Miguel Á. Carreira-Perpiñán and Weiran Wang Electrical Engineering and Computer Science University of California, Merced http://eecs.ucmerced.edu Nested

### Implementation of a Library for Artificial Neural Networks in C

Implementation of a Library for Artificial Neural Networks in C Jack Breese TJHSST Computer Systems Lab 2007-2008 June 10, 2008 1 Abstract In modern computing, there are several approaches to pattern recognition

### On the Effectiveness of Neural Networks Classifying the MNIST Dataset

On the Effectiveness of Neural Networks Classifying the MNIST Dataset Carter W. Blum March 2017 1 Abstract Convolutional Neural Networks (CNNs) are the primary driver of the explosion of computer vision.

### FAST NEURAL NETWORK ALGORITHM FOR SOLVING CLASSIFICATION TASKS

Virginia Commonwealth University VCU Scholars Compass Theses and Dissertations Graduate School 2012 FAST NEURAL NETWORK ALGORITHM FOR SOLVING CLASSIFICATION TASKS Noor Albarakati Virginia Commonwealth

### Gradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz

Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint

### Image Compression: An Artificial Neural Network Approach

Image Compression: An Artificial Neural Network Approach Anjana B 1, Mrs Shreeja R 2 1 Department of Computer Science and Engineering, Calicut University, Kuttippuram 2 Department of Computer Science and

### MoonRiver: Deep Neural Network in C++

MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement

### Neural Network Approach for Automatic Landuse Classification of Satellite Images: One-Against-Rest and Multi-Class Classifiers

Neural Network Approach for Automatic Landuse Classification of Satellite Images: One-Against-Rest and Multi-Class Classifiers Anil Kumar Goswami DTRL, DRDO Delhi, India Heena Joshi Banasthali Vidhyapith

### Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University.

Visualizing and Understanding Convolutional Networks Christopher Pennsylvania State University February 23, 2015 Some Slide Information taken from Pierre Sermanet (Google) presentation on and Computer

### 11/14/2010 Intelligent Systems and Soft Computing 1

Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in

### Residual Networks And Attention Models. cs273b Recitation 11/11/2016. Anna Shcherbina

Residual Networks And Attention Models cs273b Recitation 11/11/2016 Anna Shcherbina Introduction to ResNets Introduced in 2015 by Microsoft Research Deep Residual Learning for Image Recognition (He, Zhang,

### COMPARISION OF REGRESSION WITH NEURAL NETWORK MODEL FOR THE VARIATION OF VANISHING POINT WITH VIEW ANGLE IN DEPTH ESTIMATION WITH VARYING BRIGHTNESS

International Journal of Advanced Trends in Computer Science and Engineering, Vol.2, No.1, Pages : 171-177 (2013) COMPARISION OF REGRESSION WITH NEURAL NETWORK MODEL FOR THE VARIATION OF VANISHING POINT

### Classification: Linear Discriminant Functions

Classification: Linear Discriminant Functions CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Discriminant functions Linear Discriminant functions

### ^ Springer. Computational Intelligence. A Methodological Introduction. Rudolf Kruse Christian Borgelt. Matthias Steinbrecher Pascal Held

Rudolf Kruse Christian Borgelt Frank Klawonn Christian Moewes Matthias Steinbrecher Pascal Held Computational Intelligence A Methodological Introduction ^ Springer Contents 1 Introduction 1 1.1 Intelligent

### CS231A Course Project Final Report Sign Language Recognition with Unsupervised Feature Learning

CS231A Course Project Final Report Sign Language Recognition with Unsupervised Feature Learning Justin Chen Stanford University justinkchen@stanford.edu Abstract This paper focuses on experimenting with

### Penalizied Logistic Regression for Classification

Penalizied Logistic Regression for Classification Gennady G. Pekhimenko Department of Computer Science University of Toronto Toronto, ON M5S3L1 pgen@cs.toronto.edu Abstract Investigation for using different

### Center for Automation and Autonomous Complex Systems. Computer Science Department, Tulane University. New Orleans, LA June 5, 1991.

Two-phase Backpropagation George M. Georgiou Cris Koutsougeras Center for Automation and Autonomous Complex Systems Computer Science Department, Tulane University New Orleans, LA 70118 June 5, 1991 Abstract

### Hybrid Deep Learning for Face Verification. Yi Sun, Xiaogang Wang, Member, IEEE, and Xiaoou Tang, Fellow, IEEE

1 Hybrid Deep Learning for Face Verification Yi Sun, Xiaogang Wang, Member, IEEE, and Xiaoou Tang, Fellow, IEEE Abstract This paper proposes a hybrid convolutional network (ConvNet)-Restricted Boltzmann

### Theoretical Concepts of Machine Learning

Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5

### Convolutional Neural Networks

Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.

### Clustering. Image segmentation, document clustering, protein class discovery, compression

Clustering CS 444 Some material on these is slides borrowed from Andrew Moore's machine learning tutorials located at: Clustering The problem of grouping unlabeled data on the basis of similarity. A key

### Deep Neural Networks Applications in Handwriting Recognition

Deep Neural Networks Applications in Handwriting Recognition Théodore Bluche theodore.bluche@gmail.com São Paulo Meetup - 9 Mar. 2017 2 Who am I? Théodore Bluche PhD defended

### Day 3 Lecture 1. Unsupervised Learning

Day 3 Lecture 1 Unsupervised Learning Semi-supervised and transfer learning Myth: you can t do deep learning unless you have a million labelled examples for your problem. Reality You can learn useful representations

### Simulation of Back Propagation Neural Network for Iris Flower Classification

American Journal of Engineering Research (AJER) e-issn: 2320-0847 p-issn : 2320-0936 Volume-6, Issue-1, pp-200-205 www.ajer.org Research Paper Open Access Simulation of Back Propagation Neural Network

### Big Data Methods. Chapter 5: Machine learning. Big Data Methods, Chapter 5, Slide 1

Big Data Methods Chapter 5: Machine learning Big Data Methods, Chapter 5, Slide 1 5.1 Introduction to machine learning What is machine learning? Concerned with the study and development of algorithms that

### The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.

CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or

### Deep Reinforcement Learning

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic

### Learning internal representations

CHAPTER 4 Learning internal representations Introduction In the previous chapter, you trained a single-layered perceptron on the problems AND and OR using the delta rule. This architecture was incapable

### Programming Exercise 4: Neural Networks Learning

Programming Exercise 4: Neural Networks Learning Machine Learning Introduction In this exercise, you will implement the backpropagation algorithm for neural networks and apply it to the task of hand-written

### Decision tree learning

Decision tree learning Andrea Passerini passerini@disi.unitn.it Machine Learning Learning the concept Go to lesson OUTLOOK Rain Overcast Sunny TRANSPORTATION LESSON NO Uncovered Covered Theoretical Practical

### An Improved Backpropagation Method with Adaptive Learning Rate

An Improved Backpropagation Method with Adaptive Learning Rate V.P. Plagianakos, D.G. Sotiropoulos, and M.N. Vrahatis University of Patras, Department of Mathematics, Division of Computational Mathematics

### IMPLEMENTING DEEP LEARNING USING CUDNN 이예하 VUNO INC.

IMPLEMENTING DEEP LEARNING USING CUDNN 이예하 VUNO INC. CONTENTS Deep Learning Review Implementation on GPU using cudnn Optimization Issues Introduction to VUNO-Net DEEP LEARNING REVIEW BRIEF HISTORY OF NEURAL

### Artificial Neural Networks (Feedforward Nets)

Artificial Neural Networks (Feedforward Nets) y w 03-1 w 13 y 1 w 23 y 2 w 01 w 21 w 22 w 02-1 w 11 w 12-1 x 1 x 2 6.034 - Spring 1 Single Perceptron Unit y w 0 w 1 w n w 2 w 3 x 0 =1 x 1 x 2 x 3... x

### Machine Learning Basics. Sargur N. Srihari

Machine Learning Basics Sargur N. srihari@cedar.buffalo.edu 1 Overview Deep learning is a specific type of ML Necessary to have a solid understanding of the basic principles of ML 2 Topics Stochastic Gradient

### 6. NEURAL NETWORK BASED PATH PLANNING ALGORITHM 6.1 INTRODUCTION

6 NEURAL NETWORK BASED PATH PLANNING ALGORITHM 61 INTRODUCTION In previous chapters path planning algorithms such as trigonometry based path planning algorithm and direction based path planning algorithm