A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Size: px
Start display at page:

Download "A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems"

Transcription

1 Chapter 5 A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems 5.1 Introduction Many researchers have proposed pruning algorithms in numerous ways to optimize the network architecture (Castellano et al., 1997; Ahmmed et al., 2007; Henrique et al., 2000; Ponnapalli et al., 1999). Reed (1993) and Engelbrecht (2001) have given detailed surveys of pruning algorithms. Each algorithm has its own advantages and limitations. Some algorithms (Engelbrecht, 2001; Xing & Hu, 2009) prune both irrelevant input neurons and hidden neurons of the network and some algorithms (Zeng & Yeung, 2006) prune irrelevant hidden neurons only. Real-world applications prefer simpler and more efficient methods. But a significant drawback of most standard methods consist in their low efficiency. For example the main 94

2 weakness of the OBD and OBS techniques is their relative low computational efficiency. Magnitude based pruning (MBP) methods often remove important parts of the network as they assume that small weights are irrelevant. However small weights may be important compared to very large weights which cause saturation in hidden and output units (Engelbrecht, 2001). Some algorithms (Sietsma & Dow, 1988; Chung & Lee, 1992) require the user to specify the number of problem dependent threshold parameters or tuning parameters. More sophisticated methods (Engelbrecht, 2001; Xing & Hu, 2009; Lauret et al., 2006) reach better results, but the precision is usually compensated by unproportional increase in computation time (Reitermanova, 2008). Unfortunately, sensitivity analysis based pruning methods are not guaranteed to detect all redundant processing elements as they assume both the inputs of the network and the outputs of the hidden neurons are mutually independent (Reitermanova, 2008). When there are dependencies between inputs, the Sensitivity Analysis based method can be ineffective while the Mutual Information based methods can successfully avoid this limitation (Xing & Hu, 2009). This chapter focuses on developing a novel pruning algorithm which finds the optimal architecture of multilayer feedforward neural network (MLFNN) by removing both insignificant input nodes and hidden nodes based on a new significant measure that considers inputs of the network and outputs of the hidden neurons. The proposed work is concentrated primarily on removing nodes, since the node pruning algorithms are more efficient than weight elimination methods and the computational nodes are more important than individual connections as they rep- 95

3 resent the bottleneck through which information in a neural network is conveyed (Kruschke, 1998). This chapter is organized as follows: Section 5.2 explains the methodology of the proposed pruning, Section 5.3 describes the new pruning algorithm for optimizing the neural network architecture, Section 5.4 compares the results of the proposed method with other pruning methods in terms of pruning percentage, pruning speed and classification accuracy by implementing it on six different real datasets namely iris, Wisconsin breast cancer, hepatitis, diabetes, ionosphere and wave. 5.2 Proposed Pruning Method The basis of this pruning algorithm is to estimate the significance of each individual input node and hidden node of the trained neural network and prune all the nodes with significance value below the estimated threshold. This method is the posttraining pruning method. The proposed method, called as Neural Network Pruning by Significance (N2PS), prunes the insignificant neurons of a neural network based on its estimated significance value. This post-training pruning algorithm first uses the backpropagation training algorithm with momentum (Han & Kamber, 2001) to train a fully connected FNN and then prune the insignificant neurons Multilayer Network Model Let us consider a fully connected MLFNN as in Fig. 5.1 with an input layer, one or more hidden layers and an output layer. Let L be the total number of 96

4 Figure 5.1: MultiLayer feedforward neural network layers in a network being considered for pruning and let ml be the total number of neurons in each lth layer, 0 l L. Among L layers in a network, the first layer 0 is an input layer, the layers between 0 and L are hidden layers and the last layer L is an output layer. Let Ni0 be the ith input neuron of 0th layer where 0 i m0 and m0 th is the input neuron with bias value which is always equal to 1. Let np be the number of patterns considered for training and xip be the value of ith input neuron of pth pattern in a dataset. Let Njl be the j th neuron of lth hidden layer, 0 < l < L and 1 j ml. Let wij1 be the weight between input neuron Ni0 and neuron Nj1 in first hidden layer, where j {1, 2,..., m1 }, vjl kl+1 be the weight between a neuron Njl and neuron Nkl+1, where j {1, 2,..., ml }, k {1, 2,..., ml+1 } and initially both take random values between 1 to 1. Let hj1 be the activation value of the hidden neuron Nj1, hjl be the activation value of the hidden neuron Njl and ok be the output of the Nk th neuron in the outputl layer L, 1 k ml. The number of neurons in the output layer is equivalent to number of target classes in the dataset. Using the backpropagation algorithm, the 97

5 value of hj1 and hjl for pth pattern is calculated respectively by, hj1 = f ( m0 i=0 ml 1 (xip.wij1 )) and hjl = f ( (hjl 1.vjl 1 kl )) (5.1) jl 1 =1 where f (x) = 1,1 1+e x < l < L, and the output ok of network can be calculated by, ok = f ( ml 1 jl 1 =1 (hjl 1.vjL 1 kl )) (5.2) where f (x) = 1 1+e x The sigmoidal function f (x) = 1 1+e x is used to normalize any value within the bound 0 to 1 (Han & Kamber, 2001) Backpropagation Training The backpropagation algorithm learns iteratively by processing the np training patterns of a dataset, comparing the networks result ok for each pattern with the desired known target value dk for each target class k in a dataset. The target value is a known class label of the training pattern. Weights are modified for each pattern so as to minimize the mean squared error (mse). The value of mse is calculated according to the following equation 1 np n 1 mse =((dk ok )2 ) np p=1 k=1 2 (5.3) Weights update are made using momentum method (Setiono & Hui, 1995) in the backward direction i.e., from the output layer through hidden layer and to input layer. The purpose of inclusion of the momentum term with BP is to accelerate the convergence of the backpropagation training algorithm. The method involves 98

6 supplementing the current weight adjustments with a fraction of the most recent weight adjustment. The fraction is specified by a user selected positive momentum constant. Finally, the weights eventually converge and the learning process stops Pruning by Significance Initially, the network with the large number of hidden neurons is trained with the backpropagation with momentum algorithm to solve the classification and then the size of the trained network is optimized based on the significance of a neuron. The significance is based on its output. Equation (5.2) states that the output value of each neuron in a layer is corresponding to the sum of products of the activation value of all nodes in the previous layer and the weights of all its incoming connections. It shows that the activation value and all the outgoing weights of a node define the neuron s significance and hence the significance is computed by applying the sum-norm on those values. But before applying the sum-norm, the activation value has to be computed over the entire training set to reflect the aggregated effect of all patterns, since each pattern results in a different sensitivity matrix (Engelbrecht, 2001). To identify the aggregated activation value of a hidden neuron Njl where 0 < l < L and j {1, 2,..., ml } for all the patterns, first the total net value of each hidden neuron is identified, by computing the aggregated net values of all the patterns and then the sigmoidal function is applied on it. Applying the sigmoidal function on the input or output value of a neuron makes the neurons with larger value as closer to 1 and the neurons with smaller as closer to 0. Pruning the neurons 99

7 based only on its aggregated activation value may remove the important neurons as the significance of each node in a layer is not only based on the activation value over all the patterns but also the weights of its all outgoing connections. So the real significance of a neuron is evaluated by its aggregated activation value and also by considering the weights of its all outgoing connections. Let tnetjl be the total net value of the hidden neuron Njl and the computation of tnetjl is expressed as, tnetjl = npm0 p=1i=0 xip.wij1 ml 1 when l = 1 when 1 < l < L jl 1 =1 f (tnetjl 1 ).vjl 1 jl (5.4) After the value of tnetjl is identified, the function Sigmoidal is applied on it to compute the aggregated activation value and then the significance measure sjl of a hidden neuron Njl is computed by adding its aggregated activation value over all the patterns with all its outgoing connections as follows: sjl = ml+1 kl+1 =1 f (tnetjl ) + vjl kl+1 (5.5) where f (tnetjl ) = tnetjl1+e. 1 The status of each hidden neuron Njl, 1 l L of multilayer feedforward neural network is identified as insignificant if its significant measure sjl is smaller than the threshold value β i.e., Njl is insignif icant if sjl β, β = signif icantotherwise mljl =1 (sjl )/ml (5.6) Fig. 5.2 describes the process of identifying insignificant hidden neurons, where 100

8 all the hidden neurons with the significant measure sjl below the threshold limit are considered as insignificant neurons. Figure 5.2: Identifying insignificant hidden neurons based on threshold Similarly, the significance of an input neuron is determined by the normalized sum of all its input patterns and all its outgoing connections. The significance measure si of a node Ni0 in an input layer is computed as, si = m1 j1 =1 f (txip ) + wij1 (5.7) 1 1+e txip where 0 i < m0, f (xip ) = and txip = xip. The input neuron Ni0 is insignificant if its significant measure si is smaller than the threshold value np p=1 α i.e., Ni0 is insignif icant if si α, α = signif icantotherwise m0i=0 (si )/m0 (5.8) The threshold values α and β are calculated by finding the mean value of the significance of all nodes in that layer. After the status of each neuron is identified, 101

9 all the neurons with insignificant status are pruned from the trained neural network and then the pruned network is retrained to avoid the loss of performance due to pruning. The initial values of weights for retraining the pruned network are all inherited from the corresponding values of weights in the trained network in the previous step. After retraining, the classification accuracy of the pruned network is computed. If it falls below an acceptable level, the pruned network obtained in the previous step is retained and the process is terminated, otherwise the process is repeated. Eliminating the insignificant neurons using the threshold values α and β, calculated by the equations (5.6) and (5.8), require more pruning iterations when the number of neurons in a input layer or hidden layer is high. In this case the pruning process of that layer can be speeded up by modifying its threshold value α or β using α = q i=1 (si )/q and β = ri=1 (sjl )/r where q and r be the number of existing input neurons and the number of existing hidden neurons in lth layer, 1 < l < L in the pruned network after each pruning iteration. The main advantages of this proposed method are, (i) both insignificant input and hidden neurons are pruned, (ii) less number of retraining iterations are required since it removes insignificant neurons and it inherits the initial weights of the pruned network from the previous step, (iii) the computational cost is reduced since it doesn t require any complex calculations for pruning, (iv) the nodes are pruned directly instead of removing unwanted connections associated with those nodes, and hence the number of pruning steps is reduced, (v) No threshold or 102

10 tuning parameters are required and (vi) suitable for pruning the MLFNN with any topology. 5.3 N2PS Algorithm In this section, an algorithm which finds the optimal architecture by pruning the MLFNN based on the neuron s significance has been proposed. Input: A multilayer feedforward neural network consisting of L layers such as, an input layer with m0 input neurons, one or more hidden layers with ml hidden neurons, 0 < l < L and an output layer with ml output neurons equivalent to number of target classes, and a dataset with np patterns. Begin 1. Train the network T until a predetermined accuracy rate is achieved using the Backpropagation algorithm with momentum. 2. Consider a copy of the trained neural network T as a temporary pruned network P. 3. For each input neuron Ni0 of P in layer 0, 3.1. Compute si, the significance of the input neuron Ni0, using an equation (5.7) Frame the set I = {Ni0/(si α)}, the set of insignificant input 103

11 neurons where α = m0i=0 (si )/m0. 4. Update the temporary pruned network P by removing all the insignificant input neurons of I. 5. For each hidden neuron Njl of P in layer l, 5.1. Compute the total net value for all the patterns in a dataset using an equation (5.4) Compute sjl, the significance of the hidden neuron Njl, using an equation (5.5) Frame the set H = {Njl /(sjl β)}, the set of insignificant hidden neurons where β = mlj=1 (sjl )/ml. 6. Update the temporary pruned network P by removing all the insignificant hidden neurons of H. 7. Retrain the temporary pruned network and compute its classification accuracy on testing dataset. 8. If classification accuracy of the network P falls below an acceptable level then stop pruning otherwise consider this temporary pruned network P as trained pruned network T and goto step 2. Output: The pruned multilayer feedforward neural network T. 104

12 5.4 Experimental Results The proposed algorithm is implemented on six well known continuous and mixed mode WEKA s datasets and compared with other pruning methods such as VNP (Engelbrecht, 2001), Xing-Hu s method (Xing & Hu, 2009), MBP (Hagiwara, 1994), OBD (LeCun et al., 1990) and OBS (Hassibi et al., 1993). The datasets used to test the algorithm are, (i) Iris Plants dataset (iris), (ii) Wisconsin breast cancer dataset (cancer), (iii) Hepatitis Domain dataset (hepatitis), (iv) Pima Indians Diabetes dataset (diabetes), (v) Ionosphere dataset (ionosphere) and (vi) Wave form dataset (wave). The training and testing patterns are taken randomly from each class. For example, the iris dataset is having 3 classes with 50 patterns for each class. From each class 25 patterns are taken randomly for training and another 25 patterns are taken randomly for testing the network Result Analysis Experiments have been performed for the N2PS algorithm with the datasets namely iris, cancer, hepatitis, diabetes, ionosphere and wave. As a first step, the three layer feedforward neural network is trained with the training patterns of the dataset using the backpropagation algorithm. This algorithm uses momentum (µ) as 0.5 for all datasets and the learning rate (λ) as 0.1 for four datasets namely iris, cancer, hepatitis and diabetes and 0.9 for two datasets namely ionosphere and wave. Number of input neurons equals the number of attributes in the dataset, but one bias input is also given additionally to gain better training. The number of hid- 105

13 den neurons of the initial network are selected as completely the same with those used in (Engelbrecht, 2001) and (Xing & Hu, 2009) for comparing the performance of N2PS with the related works namely VNP and Xing-Hu s method. The network is trained until the error converges to predetermined mean squared error 0.01 or the prespecified maximum number of iterations 200 has expired, whichever is earlier. The proposed algorithm N2PS calculates the significance of each input neuron using an equation (5.5) and each hidden neuron using an equation (5.7) and it eliminates the insignificant input and hidden neurons using the equations (5.6) and (5.8) respectively. Then the pruned network is retrained as similar to training process but the prespecified maximum number of iterations is reduced to 50. Experiments were performed 10 times for each dataset by dividing the original dataset into training and testing using a different random seed every time. The average of the results of the 10 runs is calculated for each set. The performances of Table 5.1: Performance of N2PS algorithm the N2PS algorithm on six datasets are shown in Table 5.1. The results show that the algorithm doesn t require more iteration to prune the network and requires 106

14 maximum three pruning steps only. In each pruning step, the current architecture is pruned based on nodes significance and retrained. The results also show that the pruned network achieves higher accuracy than the initially selected network. The FNN with the architecture for iris dataset is trained by the Backpropagation training algorithm. It achieves the 96% classification accuracy in 120 iterations (mse=0.01). The proposed pruning algorithm N2PS removes the unwanted input neurons and hidden neurons from the trained neural network. Fig. Figure 5.3: architecture (excluding bias) pruned network of iris dataset with 98.7% accuracy 5.3 shows the pruned network of iris dataset with the classification accuracy of 98.7% for the architecture N2PS has pruned 7 hidden neurons, one input neuron and achieved higher classification accuracy. Also it finds the reduced architecture within two pruning steps and the first pruning step requires 27 iterations and the second pruning step requires only one iteration to retrain the network. On breast cancer dataset the Backpropagation training algorithm achieves the classification accuracy 95.4% in 123 iterations (mse=0.01) with the architecture feedforward neural network. The N2PS algorithm prunes the trained network 107

15 and achieves the reduced architecture with classification accuracy 97.1%. The pruned networks of this dataset are shown in Fig N2PS requires two Figure 5.4: architecture (excluding bias) pruned network of cancer dataset with 97.1% accuracy pruning steps to reduce the network and each pruning step requires only 50 iterations. The hepatitis dataset is initially trained with 25 hidden nodes. Table 5.2 shows the pruning results of the hepatitis dataset. At the first pruning step itself N2PS removes maximum number of hidden neurons i.e., out of 25 hidden neurons it removes 17. For pruning this hepatitis dataset with no reduction in accuracy, the N2PS method requires only three pruning steps. At the step of 3, N2PS has reduced the original network with architecture with accuracy 80.2% to the architecture with accuracy 86.4%. Since the performance of the network is greatly deteriorated at the step of 4, the pruning process was stopped and the current architecture is accepted. The pruning results show that the pruned network achieves higher accuracy and best generalization than the original network. The Pima Indian diabetes dataset is trained with 40 hidden nodes. In

16 Table 5.2: Pruning results of N2PS on Hepatitis dataset iterations (mse=0.14), the Backpropagation training algorithm achieves 68.6% accuracy on this dataset with the architecture feedforward neural network. The pruning procedure N2PS reduces the architecture of the trained network as with the classification accuracy 70.3%. N2PS requires only two pruning steps to reduce the network and each pruning step requires 50 iterations. The ionosphere dataset consists of 34 input attributes and 2 output classes and hence the initial architecture of the dataset is The Backpropagation algorithm trains this network up to 0.01mse and achieves 91.4% classification accuracy in 18 iterations. N2PS algorithm prunes the irrelevant input neurons and hidden neurons and finds the reduced network with the architecture for the ionosphere dataset. Also it requires only two pruning steps and the first pruning step requires 30 iterations and the second pruning step requires 41 iterations to retrain the network. The pruned network achieves the classification accuracy 94.9%. The wave-form dataset is trained with the architecture feedforward neural network. This dataset consists of 40 attributes and 3 output classes with

17 patterns. From each class 900 patterns are taken for training and the remaining patterns are taken for testing the network. In 200 iterations (mse=0.03), the Backpropagation training algorithm achieves 83.2% accuracy. The pruning procedure N2PS reduces the architecture of the trained network as Here 31 input neurons and 6 hidden neurons are removed by this algorithm and the pruned networks achieve the classification accuracy 85.5%. N2PS requires only two pruning steps to reduce the network and each pruning step requires only 50 iterations. The pruning results show that the pruned network achieves higher accuracy than the original network. The experimental results of the above examples show that the proposed method requires lesser number of pruning steps and requires lesser number of iterations for retraining the pruned network. Also the results clearly indicate that the N2PS achieves small networks with high classification accuracy and the generalization performance of the original network for all datasets are retained by the final architecture of the pruned network Comparison of Pruning Methods In this section, the performance of the proposed method is compared with other five pruning methods such as Variance Nullity Pruning (VNP), Magnitude Based Pruning (MBP), Optimal Brain Surgeon (OBS), Optimal Brain Damage (OBD) and Xing-Hu method. The pruning methods OBS and OBD require additional computation for calculating the Hessian matrix of the system but the proposed method N2PS doesn t require any complex computation to find the significant 110

18 measure of each node. The efficiency of the MBP method is also low, since it considers only the magnitude of weights to prune the network (Engelbrecht, 2001). The pruning methods OBD, OBS and MBP prune irrelevant hidden neurons only but the proposed method N2PS removes additionally the insignificant input neurons also. The sensitivity analysis based method VNP combines both the input units pruning and hidden units pruning of Multi Layer Perceptrons (MLPs) in a single formula and achieves satisfying results, but as discussed in section 1.2, VNP is not guaranteed to detect redundant neurons as it doesn t consider the mutual dependency between both the inputs of the network and outputs of the hidden neurons. The Xing-Hu s method overcomes this limitation by considering the mutual dependency between them but it performs pruning in two separate phases (Xing & Hu, 2009). The proposed method N2PS combines the advantages of both VNP and Xing-Hu. It performs pruning of the input units and hidden units of MLPs in a single formula as VNP and considers the mutual dependency between the inputs of the network and outputs of the hidden neurons like Xing-Hu s method. Xing-Hu achieves better results than VNP with two separate phases for pruning input units and hidden units respectively while N2PS achieves better results than Xing-Hu in just a single phase for pruning both units. Table 5.4 shows the comparison results of N2PS on four datasets namely iris, cancer, hepatitis and diabetes with the results of other five pruning methods in (Engelbrecht, 2001; Xing & Hu, 2009). For all pruning algorithms, a pruned network is only accepted if the deterioration in generalization is less than 1%. For all the classification problems, the proposed method resulted in better architecture with minimum number of nodes while hav- 111

19 Table 5.3: Result compariso n of N2PS with other five pruning methods 112

20 ing the accuracy similar to or better than that of other architectures obtained from other pruning methods. Regarding the classification accuracy, the N2PS algorithm achieves higher ac- curacy for all datasets except diabetes. Fig. 5.5 shows the comparison of the Figure 5.5: Comparing classification accuracies of N2PS algorithm with other pruning methods classification accuracies achieved by N2PS method and other pruning methods. It shows that the N2PS method achieves higher accuracy for all datasets than OBS, OBD and MBP and achieves maximum or equal accuracy for 3 datasets out of 4 than Xing-Hu and VNP. N2PS has also performed effectively in the removal of input neurons. Considering the removal of neurons, the N2PS method performs outstandingly while comparing with OBS, OBD and MBP and also comparable with VNP and XING-HU method. Fig. 5.6 compares the N2PS method with other pruning methods by its removal of hidden neurons. It shows that the N2PS method removes more hidden neurons for all 4 datasets than OBS, OBD and MBP and for 113

21 Figure 5.6: Comparing hidden nodes removal of N2PS with other five pruning methods 3 datasets than Xing-Hu and for 2 datasets than VNP. N2PS has also performed effectively in the removal of input neurons. Fig. 5.7 shows that the proposed algo- Figure 5.7: Comparing input nodes removal of N2PS with VNP and Xing-Hus methods rithm N2PS prunes maximum or equal input neurons for all datasets than Xing-Hu and for 3 out of 4 datasets than VNP. Considering the pruning speed of N2PS, when a network is pruned, VNP starts retraining of the reduced model on new initial random weights which may lead to the increase in number of iterations in each pruning step and decrease in classifi- 114

22 cation accuracy. But N2PS inherits the initial weights from previous step for the retraining process of the pruned network as Xing-HU (Xing & Hu, 2009). Unfortunately Xing-Hu requires more number of pruning and retraining steps for selecting the relevant input units in phase I and for removing the irrelevant hidden units in phase II. But the proposed method N2PS requires maximum 3 pruning steps only. Table 5.2 shows N2PS removes 17 hidden neurons and 10 input neurons of hepatitis dataset in a single pruning step itself. The maximum number of pruning steps required by N2PS for four data sets iris, cancer, hepatitis and diabetes respectively are 2, 2, 3 and 2 only while VNP requires 3, 7, 3 and 7. While comparing the maximum number of retraining iterations required by Xing-Hu and N2PS for the pruned network on four data sets iris, cancer, hepatitis and diabetes, Xing-Hu requires 1000, 100, 100 and 100 iterations respectively but N2PS requires only 27, 50, 50 and 50 iterations respectively. This reduction in number of pruning steps and number of retraining iterations even resulted in a better generalization than the original network and the pruned networks of the other pruning algorithms. In summary, the experimental results consistently indicate that the N2PS algorithm can reduce the neural network size significantly without reducing the network performance and hence the algorithm can be applied for the rapid removal of more irrelevant neurons from a network with large size. However when no neurons can be further removed, any weight elimination methods (Setiono & Liu 1995; Huynh & Setiono, 2005) can be used to remove single connections, for achieving more improvement in classification accuracy and optimization. 115

23 5.5 Conclusions A new pruning algorithm to determine the optimal architecture for feedforward neural network has been proposed based on new significance measure which is estimated using the Sigmoidal function and weights. Simulation results indicate that the proposed algorithm is very efficient in identifying insignificant input and hidden neurons and also confirm that the pruned neural network yields better accurate results than the original neural network used in the training phase. The main advantages of this algorithm are, no user defined parameters needs to be set, large decrease in number of nodes without affecting the classification accuracy, requires small number of pruning steps and requires small number of iterations for retraining the pruned network compared with other pruning methods and achieves better generalization ability on all datasets. The experimental results demonstrate that the proposed N2PS algorithm is very promising method for determining the optimal architecture of neural networks of arbitrary topology for classifying large datasets. 116

Reverse Engineering the Neural Networks for Rule Extraction in Classification Problems

Reverse Engineering the Neural Networks for Rule Extraction in Classification Problems Chapter 6 Reverse Engineering the Neural Networks for Rule Extraction in Classification Problems 6.1 Introduction The primary purpose of data mining is not simply classification, but to provide meaningful

More information

LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS

LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS LECTURE NOTES Professor Anita Wasilewska NEURAL NETWORKS Neural Networks Classifier Introduction INPUT: classification data, i.e. it contains an classification (class) attribute. WE also say that the class

More information

Supervised Learning in Neural Networks (Part 2)

Supervised Learning in Neural Networks (Part 2) Supervised Learning in Neural Networks (Part 2) Multilayer neural networks (back-propagation training algorithm) The input signals are propagated in a forward direction on a layer-bylayer basis. Learning

More information

Optimal Brain Damage. Yann Le Cun, John S. Denker and Sara A. Solla. presented by Chaitanya Polumetla

Optimal Brain Damage. Yann Le Cun, John S. Denker and Sara A. Solla. presented by Chaitanya Polumetla Optimal Brain Damage Yann Le Cun, John S. Denker and Sara A. Solla presented by Chaitanya Polumetla Overview Introduction Need for OBD The Idea Authors Proposal Why OBD could work? Experiments Results

More information

Notes on Multilayer, Feedforward Neural Networks

Notes on Multilayer, Feedforward Neural Networks Notes on Multilayer, Feedforward Neural Networks CS425/528: Machine Learning Fall 2012 Prepared by: Lynne E. Parker [Material in these notes was gleaned from various sources, including E. Alpaydin s book

More information

Comparative Study of Instance Based Learning and Back Propagation for Classification Problems

Comparative Study of Instance Based Learning and Back Propagation for Classification Problems Comparative Study of Instance Based Learning and Back Propagation for Classification Problems 1 Nadia Kanwal, 2 Erkan Bostanci 1 Department of Computer Science, Lahore College for Women University, Lahore,

More information

An Algorithm For Training Multilayer Perceptron (MLP) For Image Reconstruction Using Neural Network Without Overfitting.

An Algorithm For Training Multilayer Perceptron (MLP) For Image Reconstruction Using Neural Network Without Overfitting. An Algorithm For Training Multilayer Perceptron (MLP) For Image Reconstruction Using Neural Network Without Overfitting. Mohammad Mahmudul Alam Mia, Shovasis Kumar Biswas, Monalisa Chowdhury Urmi, Abubakar

More information

Neural Network Weight Selection Using Genetic Algorithms

Neural Network Weight Selection Using Genetic Algorithms Neural Network Weight Selection Using Genetic Algorithms David Montana presented by: Carl Fink, Hongyi Chen, Jack Cheng, Xinglong Li, Bruce Lin, Chongjie Zhang April 12, 2005 1 Neural Networks Neural networks

More information

Classification Lecture Notes cse352. Neural Networks. Professor Anita Wasilewska

Classification Lecture Notes cse352. Neural Networks. Professor Anita Wasilewska Classification Lecture Notes cse352 Neural Networks Professor Anita Wasilewska Neural Networks Classification Introduction INPUT: classification data, i.e. it contains an classification (class) attribute

More information

Assignment 2. Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions

Assignment 2. Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions ENEE 739Q: STATISTICAL AND NEURAL PATTERN RECOGNITION Spring 2002 Assignment 2 Classification and Regression using Linear Networks, Multilayer Perceptron Networks, and Radial Basis Functions Aravind Sundaresan

More information

Visual object classification by sparse convolutional neural networks

Visual object classification by sparse convolutional neural networks Visual object classification by sparse convolutional neural networks Alexander Gepperth 1 1- Ruhr-Universität Bochum - Institute for Neural Dynamics Universitätsstraße 150, 44801 Bochum - Germany Abstract.

More information

A Network Intrusion Detection System Architecture Based on Snort and. Computational Intelligence

A Network Intrusion Detection System Architecture Based on Snort and. Computational Intelligence 2nd International Conference on Electronics, Network and Computer Engineering (ICENCE 206) A Network Intrusion Detection System Architecture Based on Snort and Computational Intelligence Tao Liu, a, Da

More information

A Novel Technique for Optimizing the Hidden Layer Architecture in Artificial Neural Networks N. M. Wagarachchi 1, A. S.

A Novel Technique for Optimizing the Hidden Layer Architecture in Artificial Neural Networks N. M. Wagarachchi 1, A. S. American International Journal of Research in Science, Technology, Engineering & Mathematics Available online at http://www.iasir.net ISSN (Print): 2328-3491, ISSN (Online): 2328-3580, ISSN (CD-ROM): 2328-3629

More information

Classification and Regression using Linear Networks, Multilayer Perceptrons and Radial Basis Functions

Classification and Regression using Linear Networks, Multilayer Perceptrons and Radial Basis Functions ENEE 739Q SPRING 2002 COURSE ASSIGNMENT 2 REPORT 1 Classification and Regression using Linear Networks, Multilayer Perceptrons and Radial Basis Functions Vikas Chandrakant Raykar Abstract The aim of the

More information

NEURAL NETWORK-BASED SEGMENTATION OF TEXTURES USING GABOR FEATURES

NEURAL NETWORK-BASED SEGMENTATION OF TEXTURES USING GABOR FEATURES NEURAL NETWORK-BASED SEGMENTATION OF TEXTURES USING GABOR FEATURES A. G. Ramakrishnan, S. Kumar Raja, and H. V. Raghu Ram Dept. of Electrical Engg., Indian Institute of Science Bangalore - 560 012, India

More information

Simulation of Zhang Suen Algorithm using Feed- Forward Neural Networks

Simulation of Zhang Suen Algorithm using Feed- Forward Neural Networks Simulation of Zhang Suen Algorithm using Feed- Forward Neural Networks Ritika Luthra Research Scholar Chandigarh University Gulshan Goyal Associate Professor Chandigarh University ABSTRACT Image Skeletonization

More information

Image Compression: An Artificial Neural Network Approach

Image Compression: An Artificial Neural Network Approach Image Compression: An Artificial Neural Network Approach Anjana B 1, Mrs Shreeja R 2 1 Department of Computer Science and Engineering, Calicut University, Kuttippuram 2 Department of Computer Science and

More information

CS6220: DATA MINING TECHNIQUES

CS6220: DATA MINING TECHNIQUES CS6220: DATA MINING TECHNIQUES Image Data: Classification via Neural Networks Instructor: Yizhou Sun yzsun@ccs.neu.edu November 19, 2015 Methods to Learn Classification Clustering Frequent Pattern Mining

More information

Week 3: Perceptron and Multi-layer Perceptron

Week 3: Perceptron and Multi-layer Perceptron Week 3: Perceptron and Multi-layer Perceptron Phong Le, Willem Zuidema November 12, 2013 Last week we studied two famous biological neuron models, Fitzhugh-Nagumo model and Izhikevich model. This week,

More information

CHAPTER 3 RESEARCH METHODOLOGY

CHAPTER 3 RESEARCH METHODOLOGY CHAPTER 3 RESEARCH METHODOLOGY 3.1 Introduction This chapter discusses the methodology that is used in this study. The first section describes the steps involve, follows by dataset representation. The

More information

Akarsh Pokkunuru EECS Department Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

Akarsh Pokkunuru EECS Department Contractive Auto-Encoders: Explicit Invariance During Feature Extraction Akarsh Pokkunuru EECS Department 03-16-2017 Contractive Auto-Encoders: Explicit Invariance During Feature Extraction 1 AGENDA Introduction to Auto-encoders Types of Auto-encoders Analysis of different

More information

CS 6501: Deep Learning for Computer Graphics. Training Neural Networks II. Connelly Barnes

CS 6501: Deep Learning for Computer Graphics. Training Neural Networks II. Connelly Barnes CS 6501: Deep Learning for Computer Graphics Training Neural Networks II Connelly Barnes Overview Preprocessing Initialization Vanishing/exploding gradients problem Batch normalization Dropout Additional

More information

Assignment # 5. Farrukh Jabeen Due Date: November 2, Neural Networks: Backpropation

Assignment # 5. Farrukh Jabeen Due Date: November 2, Neural Networks: Backpropation Farrukh Jabeen Due Date: November 2, 2009. Neural Networks: Backpropation Assignment # 5 The "Backpropagation" method is one of the most popular methods of "learning" by a neural network. Read the class

More information

Ensemble methods in machine learning. Example. Neural networks. Neural networks

Ensemble methods in machine learning. Example. Neural networks. Neural networks Ensemble methods in machine learning Bootstrap aggregating (bagging) train an ensemble of models based on randomly resampled versions of the training set, then take a majority vote Example What if you

More information

Traffic Signs Recognition using HP and HOG Descriptors Combined to MLP and SVM Classifiers

Traffic Signs Recognition using HP and HOG Descriptors Combined to MLP and SVM Classifiers Traffic Signs Recognition using HP and HOG Descriptors Combined to MLP and SVM Classifiers A. Salhi, B. Minaoui, M. Fakir, H. Chakib, H. Grimech Faculty of science and Technology Sultan Moulay Slimane

More information

2. Neural network basics

2. Neural network basics 2. Neural network basics Next commonalities among different neural networks are discussed in order to get started and show which structural parts or concepts appear in almost all networks. It is presented

More information

A Data Classification Algorithm of Internet of Things Based on Neural Network

A Data Classification Algorithm of Internet of Things Based on Neural Network A Data Classification Algorithm of Internet of Things Based on Neural Network https://doi.org/10.3991/ijoe.v13i09.7587 Zhenjun Li Hunan Radio and TV University, Hunan, China 278060389@qq.com Abstract To

More information

The exam is closed book, closed notes except your one-page cheat sheet.

The exam is closed book, closed notes except your one-page cheat sheet. CS 189 Fall 2015 Introduction to Machine Learning Final Please do not turn over the page before you are instructed to do so. You have 2 hours and 50 minutes. Please write your initials on the top-right

More information

SNIWD: Simultaneous Weight Noise Injection With Weight Decay for MLP Training

SNIWD: Simultaneous Weight Noise Injection With Weight Decay for MLP Training SNIWD: Simultaneous Weight Noise Injection With Weight Decay for MLP Training John Sum and Kevin Ho Institute of Technology Management, National Chung Hsing University Taichung 4, Taiwan. pfsum@nchu.edu.tw

More information

COMPUTATIONAL INTELLIGENCE

COMPUTATIONAL INTELLIGENCE COMPUTATIONAL INTELLIGENCE Fundamentals Adrian Horzyk Preface Before we can proceed to discuss specific complex methods we have to introduce basic concepts, principles, and models of computational intelligence

More information

Neural Network Neurons

Neural Network Neurons Neural Networks Neural Network Neurons 1 Receives n inputs (plus a bias term) Multiplies each input by its weight Applies activation function to the sum of results Outputs result Activation Functions Given

More information

11/14/2010 Intelligent Systems and Soft Computing 1

11/14/2010 Intelligent Systems and Soft Computing 1 Lecture 7 Artificial neural networks: Supervised learning Introduction, or how the brain works The neuron as a simple computing element The perceptron Multilayer neural networks Accelerated learning in

More information

Cse634 DATA MINING TEST REVIEW. Professor Anita Wasilewska Computer Science Department Stony Brook University

Cse634 DATA MINING TEST REVIEW. Professor Anita Wasilewska Computer Science Department Stony Brook University Cse634 DATA MINING TEST REVIEW Professor Anita Wasilewska Computer Science Department Stony Brook University Preprocessing stage Preprocessing: includes all the operations that have to be performed before

More information

AMOL MUKUND LONDHE, DR.CHELPA LINGAM

AMOL MUKUND LONDHE, DR.CHELPA LINGAM International Journal of Advances in Applied Science and Engineering (IJAEAS) ISSN (P): 2348-1811; ISSN (E): 2348-182X Vol. 2, Issue 4, Dec 2015, 53-58 IIST COMPARATIVE ANALYSIS OF ANN WITH TRADITIONAL

More information

Artificial Neural Networks Lecture Notes Part 5. Stephen Lucci, PhD. Part 5

Artificial Neural Networks Lecture Notes Part 5. Stephen Lucci, PhD. Part 5 Artificial Neural Networks Lecture Notes Part 5 About this file: If you have trouble reading the contents of this file, or in case of transcription errors, email gi0062@bcmail.brooklyn.cuny.edu Acknowledgments:

More information

Artificial Neuron Modelling Based on Wave Shape

Artificial Neuron Modelling Based on Wave Shape Artificial Neuron Modelling Based on Wave Shape Kieran Greer, Distributed Computing Systems, Belfast, UK. http://distributedcomputingsystems.co.uk Version 1.2 Abstract This paper describes a new model

More information

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 6, NOVEMBER Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 6, NOVEMBER Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 6, NOVEMBER 1999 1271 Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming Bao-Liang Lu, Member, IEEE, Hajime Kita, and Yoshikazu

More information

CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS

CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of

More information

Performance Analysis of Data Mining Classification Techniques

Performance Analysis of Data Mining Classification Techniques Performance Analysis of Data Mining Classification Techniques Tejas Mehta 1, Dr. Dhaval Kathiriya 2 Ph.D. Student, School of Computer Science, Dr. Babasaheb Ambedkar Open University, Gujarat, India 1 Principal

More information

Character Recognition Using Convolutional Neural Networks

Character Recognition Using Convolutional Neural Networks Character Recognition Using Convolutional Neural Networks David Bouchain Seminar Statistical Learning Theory University of Ulm, Germany Institute for Neural Information Processing Winter 2006/2007 Abstract

More information

A neural-networks associative classification method for association rule mining

A neural-networks associative classification method for association rule mining Data Mining VII: Data, Text and Web Mining and their Business Applications 93 A neural-networks associative classification method for association rule mining P. Sermswatsri & C. Srisa-an Faculty of Information

More information

Performance analysis of a MLP weight initialization algorithm

Performance analysis of a MLP weight initialization algorithm Performance analysis of a MLP weight initialization algorithm Mohamed Karouia (1,2), Régis Lengellé (1) and Thierry Denœux (1) (1) Université de Compiègne U.R.A. CNRS 817 Heudiasyc BP 49 - F-2 Compiègne

More information

Figure (5) Kohonen Self-Organized Map

Figure (5) Kohonen Self-Organized Map 2- KOHONEN SELF-ORGANIZING MAPS (SOM) - The self-organizing neural networks assume a topological structure among the cluster units. - There are m cluster units, arranged in a one- or two-dimensional array;

More information

Neural Networks Laboratory EE 329 A

Neural Networks Laboratory EE 329 A Neural Networks Laboratory EE 329 A Introduction: Artificial Neural Networks (ANN) are widely used to approximate complex systems that are difficult to model using conventional modeling techniques such

More information

CGBoost: Conjugate Gradient in Function Space

CGBoost: Conjugate Gradient in Function Space CGBoost: Conjugate Gradient in Function Space Ling Li Yaser S. Abu-Mostafa Amrit Pratap Learning Systems Group, California Institute of Technology, Pasadena, CA 91125, USA {ling,yaser,amrit}@caltech.edu

More information

Neural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani

Neural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer

More information

Reification of Boolean Logic

Reification of Boolean Logic Chapter Reification of Boolean Logic Exercises. (a) Design a feedforward network to divide the black dots from other corners with fewest neurons and layers. Please specify the values of weights and thresholds.

More information

Back propagation Algorithm:

Back propagation Algorithm: Network Neural: A neural network is a class of computing system. They are created from very simple processing nodes formed into a network. They are inspired by the way that biological systems such as the

More information

Argha Roy* Dept. of CSE Netaji Subhash Engg. College West Bengal, India.

Argha Roy* Dept. of CSE Netaji Subhash Engg. College West Bengal, India. Volume 3, Issue 3, March 2013 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Training Artificial

More information

CS 4510/9010 Applied Machine Learning. Neural Nets. Paula Matuszek Fall copyright Paula Matuszek 2016

CS 4510/9010 Applied Machine Learning. Neural Nets. Paula Matuszek Fall copyright Paula Matuszek 2016 CS 4510/9010 Applied Machine Learning 1 Neural Nets Paula Matuszek Fall 2016 Neural Nets, the very short version 2 A neural net consists of layers of nodes, or neurons, each of which has an activation

More information

Publication A Institute of Electrical and Electronics Engineers (IEEE)

Publication A Institute of Electrical and Electronics Engineers (IEEE) Publication A Yoan Miche, Antti Sorjamaa, Patrick Bas, Olli Simula, Christian Jutten, and Amaury Lendasse. 2010. OP ELM: Optimally Pruned Extreme Learning Machine. IEEE Transactions on Neural Networks,

More information

Regularization and model selection

Regularization and model selection CS229 Lecture notes Andrew Ng Part VI Regularization and model selection Suppose we are trying select among several different models for a learning problem. For instance, we might be using a polynomial

More information

Best First and Greedy Search Based CFS and Naïve Bayes Algorithms for Hepatitis Diagnosis

Best First and Greedy Search Based CFS and Naïve Bayes Algorithms for Hepatitis Diagnosis Best First and Greedy Search Based CFS and Naïve Bayes Algorithms for Hepatitis Diagnosis CHAPTER 3 BEST FIRST AND GREEDY SEARCH BASED CFS AND NAÏVE BAYES ALGORITHMS FOR HEPATITIS DIAGNOSIS 3.1 Introduction

More information

Overlapping Swarm Intelligence for Training Artificial Neural Networks

Overlapping Swarm Intelligence for Training Artificial Neural Networks Overlapping Swarm Intelligence for Training Artificial Neural Networks Karthik Ganesan Pillai Department of Computer Science Montana State University EPS 357, PO Box 173880 Bozeman, MT 59717-3880 k.ganesanpillai@cs.montana.edu

More information

OMBP: Optic Modified BackPropagation training algorithm for fast convergence of Feedforward Neural Network

OMBP: Optic Modified BackPropagation training algorithm for fast convergence of Feedforward Neural Network 2011 International Conference on Telecommunication Technology and Applications Proc.of CSIT vol.5 (2011) (2011) IACSIT Press, Singapore OMBP: Optic Modified BackPropagation training algorithm for fast

More information

5 Learning hypothesis classes (16 points)

5 Learning hypothesis classes (16 points) 5 Learning hypothesis classes (16 points) Consider a classification problem with two real valued inputs. For each of the following algorithms, specify all of the separators below that it could have generated

More information

Deep Learning for Visual Computing Prof. Debdoot Sheet Department of Electrical Engineering Indian Institute of Technology, Kharagpur

Deep Learning for Visual Computing Prof. Debdoot Sheet Department of Electrical Engineering Indian Institute of Technology, Kharagpur Deep Learning for Visual Computing Prof. Debdoot Sheet Department of Electrical Engineering Indian Institute of Technology, Kharagpur Lecture - 05 Classification with Perceptron Model So, welcome to today

More information

Neural Network Classifier for Isolated Character Recognition

Neural Network Classifier for Isolated Character Recognition Neural Network Classifier for Isolated Character Recognition 1 Ruby Mehta, 2 Ravneet Kaur 1 M.Tech (CSE), Guru Nanak Dev University, Amritsar (Punjab), India 2 M.Tech Scholar, Computer Science & Engineering

More information

Machine Learning (CSE 446): Unsupervised Learning

Machine Learning (CSE 446): Unsupervised Learning Machine Learning (CSE 446): Unsupervised Learning Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 19 Announcements HW2 posted. Due Feb 1. It is long. Start this week! Today:

More information

Effect of the PSO Topologies on the Performance of the PSO-ELM

Effect of the PSO Topologies on the Performance of the PSO-ELM 2012 Brazilian Symposium on Neural Networks Effect of the PSO Topologies on the Performance of the PSO-ELM Elliackin M. N. Figueiredo and Teresa B. Ludermir Center of Informatics Federal University of

More information

CHAPTER 6 MODIFIED FUZZY TECHNIQUES BASED IMAGE SEGMENTATION

CHAPTER 6 MODIFIED FUZZY TECHNIQUES BASED IMAGE SEGMENTATION CHAPTER 6 MODIFIED FUZZY TECHNIQUES BASED IMAGE SEGMENTATION 6.1 INTRODUCTION Fuzzy logic based computational techniques are becoming increasingly important in the medical image analysis arena. The significant

More information

This leads to our algorithm which is outlined in Section III, along with a tabular summary of it's performance on several benchmarks. The last section

This leads to our algorithm which is outlined in Section III, along with a tabular summary of it's performance on several benchmarks. The last section An Algorithm for Incremental Construction of Feedforward Networks of Threshold Units with Real Valued Inputs Dhananjay S. Phatak Electrical Engineering Department State University of New York, Binghamton,

More information

Dr. Qadri Hamarsheh Supervised Learning in Neural Networks (Part 1) learning algorithm Δwkj wkj Theoretically practically

Dr. Qadri Hamarsheh Supervised Learning in Neural Networks (Part 1) learning algorithm Δwkj wkj Theoretically practically Supervised Learning in Neural Networks (Part 1) A prescribed set of well-defined rules for the solution of a learning problem is called a learning algorithm. Variety of learning algorithms are existing,

More information

Lecture on Modeling Tools for Clustering & Regression

Lecture on Modeling Tools for Clustering & Regression Lecture on Modeling Tools for Clustering & Regression CS 590.21 Analysis and Modeling of Brain Networks Department of Computer Science University of Crete Data Clustering Overview Organizing data into

More information

Effect of Hidden Layer Neurons on the Classification of Optical Character Recognition Typed Arabic Numerals

Effect of Hidden Layer Neurons on the Classification of Optical Character Recognition Typed Arabic Numerals Journal of Computer Science (7): 578-58, 008 ISSN 59-66 008 Science Publications Effect of Hidden Layer Neurons on the Classification of Optical Character Recognition Typed Arabic Numerals Nidal F. Shilbayeh

More information

COMPARISION OF REGRESSION WITH NEURAL NETWORK MODEL FOR THE VARIATION OF VANISHING POINT WITH VIEW ANGLE IN DEPTH ESTIMATION WITH VARYING BRIGHTNESS

COMPARISION OF REGRESSION WITH NEURAL NETWORK MODEL FOR THE VARIATION OF VANISHING POINT WITH VIEW ANGLE IN DEPTH ESTIMATION WITH VARYING BRIGHTNESS International Journal of Advanced Trends in Computer Science and Engineering, Vol.2, No.1, Pages : 171-177 (2013) COMPARISION OF REGRESSION WITH NEURAL NETWORK MODEL FOR THE VARIATION OF VANISHING POINT

More information

arxiv: v1 [cs.lg] 25 Jan 2018

arxiv: v1 [cs.lg] 25 Jan 2018 A New Backpropagation Algorithm without Gradient Descent arxiv:1802.00027v1 [cs.lg] 25 Jan 2018 Varun Ranganathan Student at PES University varunranga1997@hotmail.com January 2018 S. Natarajan Professor

More information

Inversion of Fracture Parameters by Using the Artificial Neural Network

Inversion of Fracture Parameters by Using the Artificial Neural Network IC -AI 99 International Conference 57 Inversion of Fracture Parameters by Using the Artificial Neural Network Hengchang Dai, Xiang-Yang Li, and Colin MacBeth British Geological Survey, Murchison House,

More information

Linear Separability. Linear Separability. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons

Linear Separability. Linear Separability. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons. Capabilities of Threshold Neurons Linear Separability Input space in the two-dimensional case (n = ): - - - - - - w =, w =, = - - - - - - w = -, w =, = - - - - - - w = -, w =, = Linear Separability So by varying the weights and the threshold,

More information

CS 4510/9010 Applied Machine Learning

CS 4510/9010 Applied Machine Learning CS 4510/9010 Applied Machine Learning Neural Nets Paula Matuszek Spring, 2015 1 Neural Nets, the very short version A neural net consists of layers of nodes, or neurons, each of which has an activation

More information

A new multilayer perceptron pruning algorithm for classification and regression applications

A new multilayer perceptron pruning algorithm for classification and regression applications A new multilayer perceptron pruning algorithm for classification and regression applications Philippe Thomas, Marie-Christine Suhner To cite this version: Philippe Thomas, Marie-Christine Suhner. A new

More information

Pattern Recognition 41 (2008) Contents lists available at ScienceDirect. Pattern Recognition

Pattern Recognition 41 (2008) Contents lists available at ScienceDirect. Pattern Recognition Pattern Recognition 41 (2008) 3706 -- 3719 Contents lists available at ScienceDirect Pattern Recognition journal homepage: www.elsevier.com/locate/pr Feature selection using localized generalization error

More information

Logical Rhythm - Class 3. August 27, 2018

Logical Rhythm - Class 3. August 27, 2018 Logical Rhythm - Class 3 August 27, 2018 In this Class Neural Networks (Intro To Deep Learning) Decision Trees Ensemble Methods(Random Forest) Hyperparameter Optimisation and Bias Variance Tradeoff Biological

More information

Simulation of Back Propagation Neural Network for Iris Flower Classification

Simulation of Back Propagation Neural Network for Iris Flower Classification American Journal of Engineering Research (AJER) e-issn: 2320-0847 p-issn : 2320-0936 Volume-6, Issue-1, pp-200-205 www.ajer.org Research Paper Open Access Simulation of Back Propagation Neural Network

More information

IN recent years, neural networks have attracted considerable attention

IN recent years, neural networks have attracted considerable attention Multilayer Perceptron: Architecture Optimization and Training Hassan Ramchoun, Mohammed Amine Janati Idrissi, Youssef Ghanou, Mohamed Ettaouil Modeling and Scientific Computing Laboratory, Faculty of Science

More information

Efficient Training of RBF Neural Networks for Pattern Recognition

Efficient Training of RBF Neural Networks for Pattern Recognition IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 1235 Efficient Training of RBF Neural Networks for Pattern Recognition Francesco Lampariello and Marco Sciandrone Abstract The problem

More information

A Class of Instantaneously Trained Neural Networks

A Class of Instantaneously Trained Neural Networks A Class of Instantaneously Trained Neural Networks Subhash Kak Department of Electrical & Computer Engineering, Louisiana State University, Baton Rouge, LA 70803-5901 May 7, 2002 Abstract This paper presents

More information

Improving the way neural networks learn Srikumar Ramalingam School of Computing University of Utah

Improving the way neural networks learn Srikumar Ramalingam School of Computing University of Utah Improving the way neural networks learn Srikumar Ramalingam School of Computing University of Utah Reference Most of the slides are taken from the third chapter of the online book by Michael Nielson: neuralnetworksanddeeplearning.com

More information

The Application Research of Neural Network in Embedded Intelligent Detection

The Application Research of Neural Network in Embedded Intelligent Detection The Application Research of Neural Network in Embedded Intelligent Detection Xiaodong Liu 1, Dongzhou Ning 1, Hubin Deng 2, and Jinhua Wang 1 1 Compute Center of Nanchang University, 330039, Nanchang,

More information

Network Traffic Measurements and Analysis

Network Traffic Measurements and Analysis DEIB - Politecnico di Milano Fall, 2017 Sources Hastie, Tibshirani, Friedman: The Elements of Statistical Learning James, Witten, Hastie, Tibshirani: An Introduction to Statistical Learning Andrew Ng:

More information

Forward Feature Selection Using Residual Mutual Information

Forward Feature Selection Using Residual Mutual Information Forward Feature Selection Using Residual Mutual Information Erik Schaffernicht, Christoph Möller, Klaus Debes and Horst-Michael Gross Ilmenau University of Technology - Neuroinformatics and Cognitive Robotics

More information

MATLAB representation of neural network Outline Neural network with single-layer of neurons. Neural network with multiple-layer of neurons.

MATLAB representation of neural network Outline Neural network with single-layer of neurons. Neural network with multiple-layer of neurons. MATLAB representation of neural network Outline Neural network with single-layer of neurons. Neural network with multiple-layer of neurons. Introduction: Neural Network topologies (Typical Architectures)

More information

Establishing Virtual Private Network Bandwidth Requirement at the University of Wisconsin Foundation

Establishing Virtual Private Network Bandwidth Requirement at the University of Wisconsin Foundation Establishing Virtual Private Network Bandwidth Requirement at the University of Wisconsin Foundation by Joe Madden In conjunction with ECE 39 Introduction to Artificial Neural Networks and Fuzzy Systems

More information

An Efficient Learning Scheme for Extreme Learning Machine and Its Application

An Efficient Learning Scheme for Extreme Learning Machine and Its Application An Efficient Learning Scheme for Extreme Learning Machine and Its Application Kheon-Hee Lee, Miso Jang, Keun Park, Dong-Chul Park, Yong-Mu Jeong and Soo-Young Min Abstract An efficient learning scheme

More information

Unit V. Neural Fuzzy System

Unit V. Neural Fuzzy System Unit V Neural Fuzzy System 1 Fuzzy Set In the classical set, its characteristic function assigns a value of either 1 or 0 to each individual in the universal set, There by discriminating between members

More information

EE 589 INTRODUCTION TO ARTIFICIAL NETWORK REPORT OF THE TERM PROJECT REAL TIME ODOR RECOGNATION SYSTEM FATMA ÖZYURT SANCAR

EE 589 INTRODUCTION TO ARTIFICIAL NETWORK REPORT OF THE TERM PROJECT REAL TIME ODOR RECOGNATION SYSTEM FATMA ÖZYURT SANCAR EE 589 INTRODUCTION TO ARTIFICIAL NETWORK REPORT OF THE TERM PROJECT REAL TIME ODOR RECOGNATION SYSTEM FATMA ÖZYURT SANCAR 1.Introductıon. 2.Multi Layer Perception.. 3.Fuzzy C-Means Clustering.. 4.Real

More information

6. Backpropagation training 6.1 Background

6. Backpropagation training 6.1 Background 6. Backpropagation training 6.1 Background To understand well how a feedforward neural network is built and it functions, we consider its basic first steps. We return to its history for a while. In 1949

More information

In this assignment, we investigated the use of neural networks for supervised classification

In this assignment, we investigated the use of neural networks for supervised classification Paul Couchman Fabien Imbault Ronan Tigreat Gorka Urchegui Tellechea Classification assignment (group 6) Image processing MSc Embedded Systems March 2003 Classification includes a broad range of decision-theoric

More information

Data Mining. Neural Networks

Data Mining. Neural Networks Data Mining Neural Networks Goals for this Unit Basic understanding of Neural Networks and how they work Ability to use Neural Networks to solve real problems Understand when neural networks may be most

More information

Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group

Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group 1D Input, 1D Output target input 2 2D Input, 1D Output: Data Distribution Complexity Imagine many dimensions (data occupies

More information

CHAPTER 7 MASS LOSS PREDICTION USING ARTIFICIAL NEURAL NETWORK (ANN)

CHAPTER 7 MASS LOSS PREDICTION USING ARTIFICIAL NEURAL NETWORK (ANN) 128 CHAPTER 7 MASS LOSS PREDICTION USING ARTIFICIAL NEURAL NETWORK (ANN) Various mathematical techniques like regression analysis and software tools have helped to develop a model using equation, which

More information

Univariate and Multivariate Decision Trees

Univariate and Multivariate Decision Trees Univariate and Multivariate Decision Trees Olcay Taner Yıldız and Ethem Alpaydın Department of Computer Engineering Boğaziçi University İstanbul 80815 Turkey Abstract. Univariate decision trees at each

More information

The task of inductive learning from examples is to nd an approximate definition

The task of inductive learning from examples is to nd an approximate definition 1 Initializing Neural Networks using Decision Trees Arunava Banerjee 1.1 Introduction The task of inductive learning from examples is to nd an approximate definition for an unknown function f(x), given

More information

Neural Networks (Overview) Prof. Richard Zanibbi

Neural Networks (Overview) Prof. Richard Zanibbi Neural Networks (Overview) Prof. Richard Zanibbi Inspired by Biology Introduction But as used in pattern recognition research, have little relation with real neural systems (studied in neurology and neuroscience)

More information

Perceptrons and Backpropagation. Fabio Zachert Cognitive Modelling WiSe 2014/15

Perceptrons and Backpropagation. Fabio Zachert Cognitive Modelling WiSe 2014/15 Perceptrons and Backpropagation Fabio Zachert Cognitive Modelling WiSe 2014/15 Content History Mathematical View of Perceptrons Network Structures Gradient Descent Backpropagation (Single-Layer-, Multilayer-Networks)

More information

Classification using Weka (Brain, Computation, and Neural Learning)

Classification using Weka (Brain, Computation, and Neural Learning) LOGO Classification using Weka (Brain, Computation, and Neural Learning) Jung-Woo Ha Agenda Classification General Concept Terminology Introduction to Weka Classification practice with Weka Problems: Pima

More information

Generating the Reduced Set by Systematic Sampling

Generating the Reduced Set by Systematic Sampling Generating the Reduced Set by Systematic Sampling Chien-Chung Chang and Yuh-Jye Lee Email: {D9115009, yuh-jye}@mail.ntust.edu.tw Department of Computer Science and Information Engineering National Taiwan

More information

Rough Set Approach to Unsupervised Neural Network based Pattern Classifier

Rough Set Approach to Unsupervised Neural Network based Pattern Classifier Rough Set Approach to Unsupervised Neural based Pattern Classifier Ashwin Kothari, Member IAENG, Avinash Keskar, Shreesha Srinath, and Rakesh Chalsani Abstract Early Convergence, input feature space with

More information

MLPQNA-LEMON Multi Layer Perceptron neural network trained by Quasi Newton or Levenberg-Marquardt optimization algorithms

MLPQNA-LEMON Multi Layer Perceptron neural network trained by Quasi Newton or Levenberg-Marquardt optimization algorithms MLPQNA-LEMON Multi Layer Perceptron neural network trained by Quasi Newton or Levenberg-Marquardt optimization algorithms 1 Introduction In supervised Machine Learning (ML) we have a set of data points

More information

Mini-project 2 CMPSCI 689 Spring 2015 Due: Tuesday, April 07, in class

Mini-project 2 CMPSCI 689 Spring 2015 Due: Tuesday, April 07, in class Mini-project 2 CMPSCI 689 Spring 2015 Due: Tuesday, April 07, in class Guidelines Submission. Submit a hardcopy of the report containing all the figures and printouts of code in class. For readability

More information