CLASS NOTES Models, Algorithms and Data: Introduction to computing 2018

Transcription

1 CLASS NOTES Models, Alorithms and Data: Introduction to computin 2018 Petros Koumoutsakos, Jens Honore Walther (Last update: March 5, 2018) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material (ideas, definitions, concepts, examples, etc) in these notes is taken (in some cases verbatim) for teachin purposes, from several references, and in particular the references listed below. A lare part of the lecture notes is adapted from The Nature of Mathematical Modelin by Neil Gershenfeld (Cambride University Press, 1st ed., 1998). Therefore, these notes are only informally distributed and intended ONLY as study aid for the final exam of ETHZ students that were reistered for the course Models, Alorithms and Data (MAD): Introduction to computin CONTENT: The present Notes are a first LaTex draft for the Lectures of Models, Alorithms and Data (MAD): Introduction to computin. The notes have been checked but there is no uarantee that they are free of mistakes. Please use with care. Aain, these notes are only intended for use by the ETHZ students that were reistered for the course Models, Alorithms and Data (MAD): Introduction to computin 2018 and ONLY a study aid for their final exam.

2 Chapter 2 Neural Networks 2.1 Introduction TRAINING DATA we have data {x1,y1} {xn,yn} HYPOTHESIS we make hypothesis for f f: x -> y LEARNING ALGORITHM examples so far: LSQ, steepest descent, Newton s method FINAL HYPOTHESIS Fiure 2.1: Data, hypothesis, and learnin alorithms. Example: The Netflix problem How to suest movies to a Netflix customer? We can collect the data about the customer and obtain a profile of the user (see Fiure 2.2). Then we can evaluate the similarities between all action comedy Blockbuster Tom Cruise customer movie in the data base Fiure 2.2: Profile of the customer and a iven movie in the database. movies in the database and the user s profile. The N recommended movies are the N most

3 2 Chapter 2. Neural Networks similar ones. 2.2 Learnin & functions/architectures Fiure 2.3 shows the error as a function of model complexity, i.e., number of parameters. For error under-fittin the data overfittin the data out of sample data trainin data optimal model complexity model complexity Fiure 2.3: Error as a function of model complexity for trainin and testin data. the trainin data, the error is decreasin with increased complexity. However, for the testin data (that was not used for the trainin) the error is a convex function. The minimum of this function defines the reions of under and over-fittin the data. The problem of overfittin can be checked with methods, e.., cross-validation (we split the data into fittin and testin data), bootstrappin. 2.3 Neural Network architecture There are many Neural Network (NN) types, e., forward-feed NN, convolutional NN, recurrent NN. Here, we will consider the simplest NN architecture, i.e., the fully connected forward-feed NN shown in Fiure 2.4. The leftmost layer in this network is called the input layer, and the nodes/neurons within the layer are called input nodes/neurons. The rihtmost or output layer contains the output nodes (in Fiure 2.4 a sinle output node). The middle layer is called a hidden layer. Consider a node j with I inputs x i, i = 1,.., I. The output of the node x j is iven by x j = (h j ) (2.3.1) h j = w ji x i where is the activation function and w ji are the weihts. Some of the commonly used activation functions are:

4 2.3. Neural Network architecture 3 Y NN G HK output layer node j Wkj inputs xi hj h1 hj hj hidden layer wji x x1 x2 xi xi input layer Fiure 2.4: Neuron (left) and neural network (riht) with an input layer with I input nodes, 1 hidden layer with J nodes, and an output layer with a sinle output node (K = 1). simoid: (h j ) = 1 1+e h j hyperbolic tanent : (h j ) = tanh(h j ) rectified linear units (ReLu): (h j ) = max(0, h j ) softmax or normalized exponential function: squashes a K-dimensional vector z of arbitrary real values to a K-dimensional vector σ(z) of real values in the rane [0, 1] that add up to 1, i.e., σ(z) j = for j = 1,.., K. ez j K k=1 ez k Now consider a neural network in Fiure 2.4, where the input x = (x 1, x 2,..., x I ) is a vector of size I, the hidden layer has J nodes, and the output is a vector Y NN = (Y1 NN, Y2 NN,..., YK NN) of size K, with K = 1 in our case. For nodes in the hidden layer, we can write h j = w ji x i x j = (h j ) = ( w ji x i ). (2.3.2) The value of the output node is H k = W kj x j = W kj ( w ji x i ) Y k = G(H k ). (2.3.3) We would like to train the NN on data {x l, y l }, where l = 1, 2,..., L and L is the size of the

5 4 Chapter 2. Neural Networks trainin data. The cost function E is iven by l=1 E = 1 l=l (y l Yl NN ) 2, 2 l=1 E = 1 l=l (y l G( W kj ( w ji x i ))) 2. 2 (2.3.4) The cost function can be minimized with the back-propaation method, i.e., by computin the derivative of E with respect to the weihts w, W, which are the unknown parameters of the NN. Based on the derivatives, we can update the weihts usin an optimizer. There are many optimizers, e.., radient descent, stochastic radient descent, Adam, RMSProp.