Data-analysis problems of interest

Transcription

1 Introduction 3

2 Data-analysis prolems of interest. Build computational classification models (or classifiers ) that assign patients/samples into two or more classes. - Classifiers can e used for diagnosis, outcome prediction, and other classification tasks. - E.g., uild a decision-support system to diagnose primary and metastatic cancers from gene epression profiles of the patients: Patient Biopsy Gene epression profile Classifier model Primary Cancer Metastatic Cancer 5

3 Data-analysis prolems of interest. Build computational regression models to predict values of some continuous response variale or outcome. - Regression models can e used to predict survival, length of stay in the hospital, laoratory test values, etc. - E.g., uild a decision-support system to predict optimal dosage of the drug to e administered to the patient. This dosage is determined y the values of patient iomarkers, and clinical and demographics data: Patient Biomarkers, clinical and demographics data Regression model Optimal dosage is 5 IU/Kg/week 6

4 Data-analysis prolems of interest 3. Out of all measured variales in the dataset, select the smallest suset of variales that is necessary for the most accurate prediction (classification or regression) of some variale of interest (e.g., phenotypic response variale). - E.g., find the most compact panel of reast cancer iomarkers from microarray gene epression data for, genes: Breast cancer tissues Normal tissues 7

5 Data-analysis prolems of interest 4. Build a computational model to identify novel or outlier patients/samples. - Such models can e used to discover deviations in sample handling protocol when doing quality control of assays, etc. - E.g., uild a decision-support system to identify aliens. 8

6 Data-analysis prolems of interest 5. Group patients/samples into several clusters ased on their similarity. - These methods can e used to discovery disease su-types and for other tasks. - E.g., consider clustering of rain tumor patients into 4 clusters ased on their gene epression profiles. All patients have the same pathological su-type of the disease, and clustering discovers new disease sutypes that happen to have different characteristics in terms of patient survival and time to recurrence after treatment. Cluster # Cluster # Cluster #3 Cluster #4 9

7 Basic principles of classification Want to classify ojects as oats and houses.

8 Basic principles of classification All ojects efore the coast line are oats and all ojects after the coast line are houses. Coast line serves as a decision surface that separates two classes.

9 Basic principles of classification These oats will e misclassified as houses

10 Basic principles of classification Longitude Latitude Boat House The methods that uild classification models (i.e., classification algorithms ) operate very similarly to the previous eample. First all ojects are represented geometrically. 3

11 Basic principles of classification Longitude Latitude Boat House Then the algorithm seeks to find a decision surface that separates classes of ojects 4

12 Basic principles of classification These ojects are classified as houses??? Longitude??? These ojects are classified as oats Latitude Unseen (new) ojects are classified as oats if they fall elow the decision surface and as houses if the fall aove it 5

13 The Support Vector Machine (SVM) approach Support vector machines (SVMs) is a inary classification algorithm that offers a solution to prolem #. Etensions of the asic SVM algorithm can e applied to solve prolems #-#5. SVMs are important ecause of (a) theoretical reasons: - Roust to very large numer of variales and small samples - Can learn oth simple and highly comple classification models - Employ sophisticated mathematical principles to avoid overfitting and () superior empirical results. 6

14 Main ideas of SVMs Gene Y Normal patients Cancer patients Gene X Consider eample dataset descried y genes, gene X and gene Y Represent patients geometrically (y vectors ) 7

15 Main ideas of SVMs Gene Y Normal patients Cancer patients Gene X Find a linear decision surface ( hyperplane ) that can separate patient classes and has the largest distance (i.e., largest gap or margin ) etween order-line patients (i.e., support vectors ); 8

16 Main ideas of SVMs Gene Y Cancer kernel Cancer Decision surface Normal Normal Gene X If such linear decision surface does not eist, the data is mapped into a much higher dimensional space ( feature space ) where the separating decision surface is found; The feature space is constructed via very clever mathematical projection ( kernel trick ). 9

17 Necessary mathematical concepts

18 How to represent samples geometrically? Vectors in n-dimensional space (R n ) Assume that a sample/patient is descried y n characteristics ( features or variales ) Representation: Every sample/patient is a vector in R n with tail at point with coordinates and arrow-head at point with the feature values. Eample: Consider a patient descried y features: Systolic BP and Age 9. This patient can e represented as a vector in R : Age (, 9) (, ) Systolic BP

19 How to represent samples geometrically? Vectors in n-dimensional space (R n ) Patient 3 Patient 4 6 Age (years) 4 Patient Patient Patient id Cholesterol (mg/dl) Systolic BP (mmhg) Age (years) Tail of the vector Arrow-head of the vector 5 35 (,,) (5,, 35) 5 3 (,,) (5,, 3) (,,) (4, 6, 65) (,,) (3, 8, 45) 3

20 How to represent samples geometrically? Vectors in n-dimensional space (R n ) Patient 3 Patient 4 6 Age (years) 4 Patient Patient Since we assume that the tail of each vector is at point with coordinates, we will also depict vectors as points (where the arrow-head is pointing). 4

21 Purpose of vector representation Having represented each sample/patient as a vector allows now to geometrically represent the decision surface that separates two groups of samples/patients. 7 A decision surface in R A decision surface in R In order to define the decision surface, we need to introduce some asic math elements 5 5

22 Hyperplanes as decision surfaces A hyperplane is a linear decision surface that splits the space into two parts; It is ovious that a hyperplane is a inary classifier. A hyperplane in R is a line A hyperplane in R 3 is a plane A hyperplane in R n is an n- dimensional suspace 3

23 Equation of a hyperplane Consider the case of R 3 : P w P An equation of a hyperplane is defined y a point (P ) and a perpendicular vector to the plane ( ) at that point. w O Define vectors: and, where P is an aritrary point on a hyperplane. OP OP A condition for P to e on the plane is that the vector is perpendicular to : w ( ) w w w + or w define w The aove equations also hold for R n when n>3. 34

24 Equation of a hyperplane Eample w (4,,6) P (,, 7) w ( 4) 43 w + 43 (4,,6) + 43 (4,,6) ( 4 () () (), + 6 () (3), (3) ) direction P - direction w w + 5 w + 43 w + What happens if the coefficient changes? The hyperplane moves along the direction of w. We otain parallel hyperplanes. Distance etween two parallel hyperplanes is equal to. D / w and w + w + 35

25 (Derivation of the distance etween two parallel hyperplanes) 36 w + w + w w w t D w t w t w t w w t w tw w w w t tw D tw / ) / ( ) ( ) ( w t

26 Recap We know How to represent patients (as vectors ) How to define a linear decision surface ( hyperplane ) We need to know How to efficiently compute the hyperplane that separates two classes with the largest gap? Gene Y Need to introduce asics of relevant optimization theory Normal patients Cancer patients Gene X 37

27 Case : Linearly separale data; Hard-margin linear SVM Given training data: Negative instances (y-) y,, y,...,,..., y N N R Positive instances (y+) n {, + } Want to find a classifier (hyperplane) to separate negative instances from the positive ones. An infinite numer of such hyperplanes eist. SVMs finds the hyperplane that maimizes the gap etween data points on the oundaries (so-called support vectors ). If the points on the oundaries are not informative (e.g., due to noise), SVMs will not do well. 47

28 Statement of linear SVM classifier w + Negative instances (y-) Since we want to maimize w the gap, we need to minimize or equivalently minimize w + w + Positive instances (y+) + The gap is distance etween parallel hyperplanes: w + w + + Or equivalently: We know that Therefore: and w + ( + ) w + ( ) D D / w ( is convenient for taking derivative later on) w / w 48

29 Statement of linear SVM classifier w + w + w + + In addition we need to impose constraints that all instances are correctly classified. In our case: w w i i if if y i y i + Equivalently: y ( w + ) i i Negative instances (y-) Positive instances (y+) In summary: w yi ( w i + ) f ( ) sign( w + ) Want to minimize suject to for i,,n Then given a new instance, the classifier is 49

30 Case : Not linearly separale data; Soft-margin linear SVM What if the data is not linearly separale? E.g., there are outliers or noisy measurements, or the data is slightly non-linear. Want to handle this case without changing the family of decision functions. Approach: ξ i Assign a slack variale to each instance, which can e thought of distance from the separating hyperplane if an instance is misclassified and otherwise. N w + C ξ Want to minimize suject to for i,,n i Then given a new instance, the classifier is i yi ( w i + ) ξi f ( ) sign( w + ) 55

31 Case 3: Not linearly separale data; Kernel trick Gene Tumor kernel Tumor? Φ? Normal Normal Gene Data is not linearly separale in the input space Data is linearly separale in the feature space otained y a kernel Φ : R N H 58

32 Eample of enefits of using a kernel 4 () 3 () Data is not linearly separale in the input space (R ). Apply kernel K(, z) ( z) to map data to a higher dimensional space (3- dimensional) where it is linearly separale. K(, z) ( z) () z () + () z () () z () () () z z + () () () z () [ z + z ] () () () () () () () () z z z () () () z () Φ( ) Φ( z) 65

33 Eample of enefits of using a kernel Therefore, the eplicit mapping is Φ( ) () () () () () () kernel, 4 3 () Φ 3, 4 () () () 66