Machine Learning in Biology

Transcription

1 Università degli studi di Padova Machine Learning in Biology Luca Silvestrin (Dottorando, XXIII ciclo)

2 Supervised learning Contents Class-conditional probability density Linear and quadratic discriminant (parametric) k Nearest Neighbor (non parametric) Class boundaries Decision trees Support Vector Machines Neural networks Unsupervsed Learning Machine Learning in Biology 2

3 Introduction A brain possesses many advantages over a digital computer: Human brain Computer Superior number crunching capabilities It quickly recognizes faces, even in bad light and in an environment full of other objects It understands speech, even of an unknown person and in a noisy environment It doesn t stop working just because a few cells die It can learn from experience Normally it doesn t survive any CPU degradation It requires new software or software upgrade Machine Learning in Biology 3

4 Learning Given a separable problem, we can find suitable values of the weights and the thresholds of the network in an iterative manner, presenting to the network examples with a known classification. This is called training or learning. Two main paradigms on training exist: supervised and unsupervised learning. Supervised learning: objects in a given collection are classified using a set of attributes, or features. The result of the classification process is a set of rules that prescribe assignments of objects to classes based solely on values of features. Examples in biology are: - genetic code -- protein secondary structure - tissue gene expression profiles -- disease group Unsupervised learning no predefined class labels are available for the objects under study. In this case, the goal is to explore the data and discover similarities between objects. Similarities are used to define groups of objects, referred to as clusters. Examples: -gene expression data classify and to identify new disease groups -genetic code -- protein secondary structure Machine Learning in Biology 4

5 Supervised learning General case - {x 1, x 2, x n } a set of n data - x ii measured value of the variable (feature) j of object (sample) i. - X = (x ij ) n x p matrix - K (existing) classes - y i = 1, 2, c, K class label associated to x i In such multiclass classification problems, a classifier C(x) may be viewed as a collection of K discriminant functions g c (x) such that the object with feature vector x will be assigned to the class c for which g c (x) is maximized over the class labels c in {1,...,K}. The feature space X is thus partitioned by the classifier C(x) into K disjoint subsets. But in general C(x) is unknown, and needs to be estimated from a set of correctly classified samples: the training or design set. Machine Learning in Biology 5

6 Discriminant function indentification There are two main approaches to the identification of the discriminant functions g c (x). The first assumes knowledge of the underlying class-conditional probability density function: the object x will be classified in the class in which it has the higher membership probability. (linear and quadratic discriminants, k-nearest neighbor decision rule) The second approach is to use data to estimate the class boundaries directly, without explicit calculation of the probability density functions. (decision trees, neural networks, and support vector machines) Machine Learning in Biology 6

7 Error The confusion matrix A confusion matrix contrasts the class labels attributed to the object by the trained classifier with their true class: Example: 2 classes; dataset: 100 elements. predicted 1 2 true The error rate is defined as the average number of misclassified samples: the off diagonal elements of the matrix (in percent respect the total number of elements). Error rate = 30% The accuracy is the fraction of samples correctly classified: 1 ErrRate = 70% Machine Learning in Biology 7

8 Validation If the data used to build the classifier is also used to compute the error rate (resubstitution), then it will be optimistically biased. There are better ways to assess the error: - hold-out procedure: data split in two halves: the training set and the test set - leave-one-out (LOO) cross validation method: the classifier is trained n times on (n-1) data. Gives low bias, but may show high variance. - N fold cross validation method: (n-m) training points and m test points. Sampling in different ways you can obtain mean and standard deviation of the classifier error. Machine Learning in Biology 8

9 Quadratic and linear discriminant This method is suitable for continuous variables, and is used in the study of gene expression data. We assume that for each class c, x follows the multivariate normal distribution N(m c, Σ c ), having a mean m c and a covariance matrix Σ with dimensions p x p. The discriminant function for each class can be computed as: g c (x) = - (x m c ) Σ c -1 (x m c ) T log( Σ c ) where the class mean m c and covariance matrix Σ c are sample-derived estimates. - If the number of samples per class is low, one can assume the K matrixes all the same. The resulting classifier uses hyperplanes as class boundaries, hence the name normal-based linear discriminant. - When the number of features is comparable with the number of samples, a further simplification can be made, by setting all off-diagonal elements in the covariance matrix to zero (between-features covariation is disregarded). Such a diagonal linear discriminant was found to outperform other types of classifiers on a variety of microarray analyses. Machine Learning in Biology 9

10 Linear discriminant analisys: examples Machine Learning in Biology 10

11 Quadratic and linear discriminant comparison Machine Learning in Biology 11

12 K Nearest Neighbor classifier The k-nn classifier: nonparametric method of density estimation, it implies the continuity of the feature variables (no assumption on the data distribution). It does not require model fitting, but simply stores the training dataset with all available vector prototypes of each class. To classify a new object z the algorithm: - computes the distance (e.g. Euclidean) between z and all the available objects in the training set, xi, i = 1,...,n. - the discriminant function g c (x) attributes to z to the class of of the top k closest samples Machine Learning in Biology 12

13 K Nearest Neighbor classifier: examples k nearest neighbor classifier (k=3) vs quadratic classifier k nearest neighbor classifier (k=8) knn java applet: matrix size 80 x 80 samples 1000 top right picture k = 2, ErrRate = 5.5% bottom right picture k = 20, ErrRate = 7.2% Machine Learning in Biology 13

14 Decision tree Three main steps: 2. Selecting a splitting rule: a feature and a threshold to partition the data set. 3. Determining the terminal nodes. 4. Assigning class labels to terminal nodes. It is trained by an iterative selection of a individual features that are most salient at each node in the three. Machine Learning in Biology 14

15 Support vector machine Suitable for linearly separable classification problems. SVM finds an optimal hyperplane wx T + b = 0 w is the p-dimensional vector perpendicular to the hyperplane, b is the bias. such that the maximum margin between the two classes is achieved. The margin is defined as the distance between the decision hyperplane and the closest training samples (1/ w 2 ). Applying a kernel transformation, SVM can be applied even to separable problems where more sophisticated (non linear) decision boundaries are required. Machine Learning in Biology 15

16 Neural network The computations of the brain are done by a highly interconnected network of neurons, which communicate by sending electric pulses through the neural wiring consisting of axons, synapses and dendrites. McCullock and Pitts model, If the total input is above a threshold, the output of the unit is 1, otherwise 0. Machine Learning in Biology 16

17 Perceptrons The simplest model of a perceptron. Simple perceptrons could solve only the very limited class of linear separable problems. It behaves like a linear classifier. Machine Learning in Biology 17

18 Multi layer networks It was showed that more complex network, trained with error back propagation method, could solve non linear problems, like the xor function. xor B nb A 0 1 na 1 0 Right: Multi-layer perceptron or feed-forward network. The introduction of a hidden layer of threshold units adds more hyperplanes in the space. Multi-layer networks are used to classify tumors (NETtalk). Machine Learning in Biology 18

19 Network training All weights are set to small random number. Total error is estimated: Σ i=1 n (y i calc. y i req. ) 2 y i calc. = calculated class; y i req. = required class; The weights and thresholds are changed each time an example is presented. The process is iterated until the error no longer changes. A technique to optimize the training process is the gradient descent (but it s not garanteed to find the global minimum in error landscape). Machine Learning in Biology 19

20 Over-fitting problem Overfitting occurs when the network has too many parameters to be learnt. This is true for every method of classification or regression, it has got a particular relevance for neural networks. A network that overfits training data in unlikly to generalize well for new data. Different cross validation methods can be used to validate the ability of the network. Overfitting example: the lines refers to Neural Networks with different numbers of hidden units: Green: 1 hidden unit Blue: 10 hidden units Purple: 20 hidden units. The points are fitted perfectly, but in the intermediate regions the NN is overly creative. Machine Learning in Biology 20

21 Conclusions Modern biology largely benefits from the advancements made in the field of Machine Learning: we still cannot predict structure for any protein sequence, but we have come closer, and growing databases facilitate the task. But caution must be used in the judgement of the results reached in this way. All steps involved in the classifier design should be cross validated to obtain an unbiased estimate for classifier accuracy. (Many studies published in literature as successful have been showed to be overoptimistic). Choice, tuning and diagnosis of machine learning applications are far from mechanical. Machine Learning in Biology 21

22 Bibliography Neural networks predict protein structure: hype or hit?, B Rost, in: 'Artificial intelligence and heuristic methods in bioinformatics (eds. P Frasconi, R Shamir), 2003: What are artificial neural networks?, A Krogh, Nature Biotech 26, 195 (2008) Machine Learning and its application to Biology, AL Tarca et al, PLoS Comput Biol, 3, e116 (2007) Machine Learning in Biology 22