A Quantitative Comparison of Neural Networks and Logistic Regression

Transcription

1 A Quantitative Comparison of Neural Networks and Logistic Regression Lakshay Piplani 1, Kartikeya Gupta 2, Ekansh Goyal 3 Student, Maharaja Agrasen Institute of Technology, GGSIPU 1,2,3, India ABSTRACT: In this research paper, we aim to quantitatively compare two popular classification methods - Logistic Regression and Neural Networks. We use the one-vs-all classification approach for a multiclass classification problem, wherein we aim to develop prediction models for hand-written digit (from 0 to 9) recognition. Automated handwritten digit recognition has varied application in today s world ranging from reading numbers written on bank cheques to reading postal codes. Our results show that the Neural Network model is able to more accurately identify digits than the Logistic Regression model. INTRODUCTION Machine Learning is the science of getting computers to act without explicitly programming them. As a formal definition, Machine Learningis a branch of computer science that evolved from the study of pattern recognition and computational learning theory in artificial intelligence.it uses data to produce predictive models that make predication on unknown data. Machine learning has given us the wonders of self-driving cars, practical speech recognition, effective web search, spam filtering, and has had a profound effect on our daily lives. [1] Some of the problems in the realm of Machine Learning include regression, classification, clustering, and so forth.the focus of this paper will be on classification using Logistic Regression and Neural Network. LOGISTIC REGRESSION Logistic regression(lr, henceforth) is a supervised machine learning technique used for classification purposes.lr has wide applicability in a variety of fields such as computer vision, marketing, social sciences, and so forth. It is well-known for its simplicity and delivers great predictive value for the effort it requires. LR models return the probability that a data point belongs to a certain class. The probability is then mapped to either 0 or 1 (in a 0-1 classification), depending on whether the probability is below or above 0.5, whereas in a standard regression model, the outcomes consist of a continuous range of real numbers. More formally, for a sample xi Rp whose label is denoted as yi, the probability of yi being positive is predicted to be P{yi= +1} = 1, 1+e β>xi, given the LR model parameter β. In order to obtain a parameter that performs well, often a set of labelled samples{(x1, y1),...,(xn, yn)} are collected to learn the LR parameter β which maximizes the induced likelihood function over the training samples. [2] NEURAL NETWORK The concept of a Neural Network (NN henceforth) is basically introduced from the subject of biology where neural network plays an important and key role in human body. NNs mimic the human brain in structure and functioning. The fundamental unit in an NN is similar to the neurons in human brain. NN is just a web of inter connected neurons which are millions and millions in number. These units are functionally the same as those used in LR. An Artificial Neuron is basically an engineering approach of biological neuron. Just like the neurons in a human brain are connected to each other, the computational units in an NN are also connected to each other in a pre-determined fashion. These units are arranged in layers. An NN typically has an input layer, a few hidden layers, and an output layer. These layers are connected in a cascaded fashion. The data points are input at the input layer, processed at the hidden layers, and the prediction is output at the output layer. To move from one layer to another, we use a matrix of parameters. Each layer, from the input to the last hidden layer, has a matrix of parameters associated with it to traverse to the next layer. [3]

2 forecasting and monitoring of snow, ice and forests [7]. Fig1 Fig 2 These two algorithms have wide range of utilities and used in almost all problems of machine learning in order to get an optimized hypothesis function so as to predict the outcome of data These algorithms were implemented to compare the landscape susceptibility in Kat country(tokat- Turkey). Landscape related factors such as geology, faults, drainage system, slope angle, slope aspect and stream power index(spi) were used in it. The accuarcy of susceptibility map shows neural network method is more accurate than logistic regression [4] These were also used in building credit scoring models in credit union environment. Their results indicate that customized neural networks offer a very promising avenue if the measure of performance is percentage of bad loans correctly classified. However, if the measure of performance of good and bad loans correctly, logistic regression models are comparable to the neural networks approach. [5] These algorithms are appearing as useful alternatives to traditional statistical modelling techniques in many scientific disciplines also. [6]. They were implemented in atmospheric sciences also for weather and climate prediction, air quality forecasting, oceanographic and hydrological These algorithms also proves to be a valuable tool for marketers concerned with predicting customer choice that is marketing science. It has been shown that neural network provide superior predictions regarding customer decision processes than any other machine learning algorithm. The reason for the same is that neural network models can offer significant improvement over traditional statistical methods because of their ability to capture nonlinear relationships associated with the use of non compensatory decision rules. [8]. Prediction of firm bankruptcies have been extensively studied in accounting, as all stakeholders in a firm have a vested interest in monitoring its financial performance. These algorithms have been used to compare the predictive capabilities for firm bankruptcy showing that neural network again perform better in predicting firm bankruptcies.[9]. METHODS: Our dataset consists of 5000 training examples, where each training example is a 20 pixel by 20 pixel greyscale image of the digit. Each pixel is represented by a floating point number indicating the greyscale intensity at that location. The 20 by 20 grid of pixels is unrolled" into a 400-dimensional feature vector. Each of these training examples becomes a single row in our data matrix X. This gives us a 5000 by 400 matrix X where every row is a training example for a handwritten digit image. The second part of the training set is a 5000-dimensional vector y that contains labels for the training set. This dataset was obtained courtesy of MNIST handwritten digit dataset. In all further references and equations, m is the number of training examples, Θ denotes the learning parameters or weights that are applied to data points, λ is the regularization parameter, and h is the difference between the actual output and the prediction.we chose MATLAB as our development platform as it renders itself naturally to matrix algebra, which we used extensively during our research. Each of the two models are trained on randomly selected (without repetition) training sets of sizes

3 varying from 2500 to randomly selected (without repetition) data points are set aside as the test data set. In order to maintain consistency, the same test set is used for testing the models. The training and test set errors obtained for different training set sizes are used to plot learning curves. THE LOGISTIC REGRESSION MODEL: First, the Logistic Regression model was developed, which was used to predict on the test dataset. The procedure for training the model is as follows: 1. Calculate the regularized logistic regression cost function and gradients. The regularized logistic regression cost function is given by: The regularized logistic regression gradient is given by: The following is an excerpt from our code: htheta = sigmoid(x*theta); J = sum( (-y.*log(htheta)) - ((1-y).*log(1-hTheta)) ); J = J/m; regsum = sum(theta(2:end).^2) * lambda / (2 * m); J = J + regsum; error = htheta-y; grad = X' * error; grad = grad / m; thetagrad = theta; thetagrad(1) = 0; thetagrad = thetagrad * (lambda/m); grad = grad + thetagrad; 2. Training models for one-vs-all classification: We train ten different models for 10 digits, and store the parameters so obtained in a two-dimensional matrix, each row corresponding to the learned parameters for one class. We initialize all the parameters to zero and use the fmincg function in MATLAB to obtainparameters for minimized cost function. for c=1:num_labels initial_theta = zeros(n + 1, 1); options = optimset('gradobj', 'on', 'MaxIter', 50); [theta] =...fmincg (@(t)(lrcostfunction(t, X, (y == c), lambda)),...initial_theta, options); all_theta(c, 1:end) = theta(:)' ; 3. Prediction on test set After training the models, we use them to predict on the test data set and calculate the prediction accuracy. For each data point, we calculate the probability that it belongs to a certain class and then chose the class for which the probability is the highest. The label of this class is returned as the prediction for that data point. pred = sigmoid(all_theta * X'); i = 0; [~,i] = max(pred, [], 1); THE NEURAL NETWORK MODEL: Theprocedure for development of the Neural Network model is shown next. We chose a simple Neural Network model that has one input layer, one hidden layer, and one output layer. The input layer has 400 units, the second layer 25 and the output layer 10 units, all excluding the bias units. The steps for training and testing the model are as follows: 1. Feedforward and regularized Neural Network cost function. We implement the feedforward propagation function that will be used to predict on the test data set. An excerpt from our code is as follows: layer2 = sigmoid(theta1 * X'); layer2 = [ones(1,m) ; layer2];

4 outputlayer = sigmoid(theta2*layer2); i=0; [~,i] = max(outputlayer, [],1); The regularized Neural Network cost function is given by: z2 = X * Theta1'; z2sig = sigmoid(z2); z2sig = [ones(m,1) z2sig]; z3 = z2sig * Theta2'; htheta = sigmoid(z3); ytemp = 0; htemp = 0; sum1=0; fori = 1:m, ytemp = 1:num_labels; htemp = 1:num_labels; ytemp = (ytemp == y(i)); htemp = htheta(i, 1:end); sum1 = sum1 + sum((-ytemp).*log(htemp) - (1- ytemp).*log(1- htemp)); sum1 = sum1/m; sum2 = 0; sum3 = 0; Theta1Squared = Theta1.^2; Theta1Squared = Theta1Squared(1:end, 2:end); Theta1Squared = Theta1Squared(:); sum2 = sum(theta1squared); Theta2Squared = Theta2.^2; Theta2Squared = Theta2Squared(1:end, 2:end); Theta2Squared = Theta2Squared(:); sum3 = sum(theta2squared); sum4 = sum1 + lambda*(sum2 + sum3)/(2*m); J=sum4; The parameters are given small random initial values instead of initializing them to zero. As before, we use the fmincg advanced opitmizer in MATLAB to obtain a good set of learning parameters and minimize the cost function. A gradient checking step is also performed to ensure that the gradients obtained through backpropagation are indeed correct. Delta1=0; Delta2=0; for t=1:m, ytemp = 1:num_labels; ytemp = (ytemp == y(t)); htemp = htheta(t, 1:end); z2temp = z2(t, 1:end)'; del3 = (htemp-ytemp)'; del2 = (Theta2(1:end, 2:end)' * del3 ).*sigmoidgradient(z2temp); Delta1 = Delta1 + (del2 * X(t, 1:end)); Delta2 = Delta2 + (del3 * z2sig(t, 1:end)); Theta1_grad = Delta1/m; Theta2_grad = Delta2/m; Theta1Temp = [zeros(size(theta1, 1),1) Theta1(1:end, 2:end)]; Theta2Temp = [zeros(size(theta2, 1),1) Theta2(1:end, 2:end)]; Theta1_grad = Theta1_grad + lambda*(theta1temp)/m; Theta2_grad = Theta2_grad + lambda*(theta2temp)/m; 3. Prediction on test set The model so obtained is used to predict on the test set. The label of the class with highest probability is returned as the prediction. An excerpt from our code is as follows: outputlayer = sigmoid(theta2*layer2); i=0; [~,i] = max(outputlayer, [],1); 2. Backpropagation to compute gradients of the cost function RESULTS: The learning curves are shown below:

5 REFERENCES: [1]P. Domingos, 'A few useful things to know about machine learning', Communications of the ACM, vol. 55, no. 10, p. 78, [2]G. Taylor and M. Becker, 'Increased efficiency of analyses: cumulative logistic regression vs ordinary logistic regression', Community Dentistry and Oral Epidemiology, vol. 26, no. 1, pp. 1-6, [3]J. Castro, C. Mantas and J. Benitez, 'Interpretation of artificial neural networks by means of fuzzy rules', IEEE Trans. Neural Netw., vol. 13, no. 1, pp , [4]I. Yilmaz, 'Landslide susceptibility mapping using frequency ratio, logistic regression, artificial neural networks and their comparison: A case study from Kat landslides (Tokatâ Turkey)', Computers & Geosciences, vol. 35, no. 6, pp , [5]V. Desai, J. Crook and G. Overstreet, 'A comparison of neural networks and linear scoring models in the credit union environment', European Journal of Operational Research, vol. 95, no. 1, pp , [6]M. Gardner and S. Dorling, 'Artificial neural networks (the multilayer perceptron)â a review of applications in the atmospheric sciences', Atmospheric Environment, vol. 32, no , pp , [7]W. Hsieh, Machine learning methods in the environmental sciences. Cambridge, UK: Cambridge University Press, [8]P. West, P. Brockett and L. Golden, 'A Comparative Analysis of Neural Networks and Statistical Methods for Predicting Consumer Choice', Marketing Science, vol. 16, no. 4, pp , [9]R. Wilson and R. Sharda, 'Bankruptcy prediction using neural networks', Decision Support Systems, vol. 11, no. 5, pp , From the learning curves, it is observed that the test set prediction error decreases as we increase the size of the training set. The crux of our findings is that the average prediction error of the Neural Network is less than the average prediction error of the Logistic Regression model.this confirms the fact that Neural Networks, being more complex in nature and hence of better predictive value, are able to deliver nonlinear and more complex models as compared to Logistic Regression technique