NEURAL NETWORKS ... FEATURE SELECTION USING ANT COLONY OPTIMIZATION: APPLICATIONS IN HEALTH CARE. Motivation. Outline.

Transcription

1 Motivation FEATURE SELECTION USING ANT COLONY OPTIMIZATION: APPLICATIONS IN HEALTH CARE João M. C. Sousa S. M. Vieira, S. N. Finkelstein 2,3, A. S. Fialho,2, F. Cismondi,2, S. R. Reti 3 and M. D. Howell 3 Knowledge discovery process Data acquisition Data Preprocessing Target data Feature selection Preprocessed data Modeling Reduced data Interpretation Patterns Knowledge Technical University of Lisbon, Instituto Superior Técnico, Dept. of Mechanical Engineering, CIS/IDMEC LAETA, Av. Rovisco Pais, 49- Lisbon, Portugal 2 Massachusetts Institute of Technology, Engineering Systems Division, 77 Massachusetts Avenue, 239 Cambridge, MA, USA 3 Division of Clinical Informatics, Department of Medicine, Beth Israel Deaconess Medical Centre, Harvard Medical School, Boston, MA, USA From G. Piatetsky-Shapiro U. Fayyad and P. Smyth. From data mining to knowledge discovery in databases. Artificial Intelligence Magazine, 7(3):37-54, September 2 Eindhoven, the Netherlands 2 Outline Motivation Modeling Neural networks Fuzzy sets and systems Fuzzy modeling Feature selection Ant colony optimization Ant feature selection Application: predicting outcomes of sepsis patients NEURAL NETWORKS 2 September 2 Eindhoven, the Netherlands 3 Artificial neuron Types of neurons x x 2... w 2 Neuron y McCulloch and Pits (943) Threshold : n y sign wx i i i Other types of activation functions (net = w i x i ) x n w n x i : i-th input of the neuron w i : synaptic strength (weight) for x i y = (w i x i ): output signal step, if net y, if net y net e sigmoid y linear net Eindhoven, the Netherlands 5 Eindhoven, the Netherlands 6

2 Multi-Layer Perceptron (MLP) Most common MLP Can learn functions that are not linearly separable. Output signals x... x i... x n h b h w h w ij w nm h b m h Hidden layer 2... j... m o o w b o w jk o w ml o b l Output layer... k... l y y k y l Eindhoven, the Netherlands 7 Eindhoven, the Netherlands 8 Most common MLP Learning in NN Output of neurons in the hidden-layer h j : tanh n n i i h n h wx i ij i h wxb wx j ij i j ij i Output of neurons in the output-layer y k : m m o o o j j y w h b w h k jk j j jk k m j w h o jk j sigmoid linear Biological neural networks: Synaptic connections amongst neurons which simultaneously exhibit high activity are strengthned. Artificial neural networks: Mathematical approximation of biological learning. Error minimization (nonlinear optimization problem). Error backpropagation (first-order gradient) Newton methods (second-order gradient) Levenberg-Marquardt (second-order gradient) Conjugate gradients... Eindhoven, the Netherlands 9 Eindhoven, the Netherlands Supervised learning Bibliography x Training data: e y T T T T 2 N X x x x T T T T 2 N Y y y y S. Haykin. Neural Networks - A Comprehensive Foundation. Prentice Hall, 999. J.-S. Jang, C.-T. Sun and E. Mizutani. Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence. Prentice Hall, New Jersey, 997. Andries P. Engelbrecht. Computational Intelligence: An Introduction. John Wiley, Chichester, 22 Michael Negnevitsky. Artificial Intelligence: A Guide to Intelligent Systems. Addison-Wesley, Pearson Education, 22. Eindhoven, the Netherlands Eindhoven, the Netherlands 2 2

3 Introduction FUZZY SETS Basic Concepts How to simplify very complex systems? Allow some degree of uncertainty in their description! How to deal mathematically with uncertainty? Using probabilistic theory (stochastic). Using the theory of fuzzy sets (non-stochastic). Proposed in 965 by Lotfi Zadeh (Fuzzy Sets, Information Control, 8, pp ). Imprecision or vagueness in natural language does not imply a loss of accuracy or meaningfulness! Eindhoven, the Netherlands 4 Classical set Logic propositions Example: set of old people A = {age age 7} A Nick is old... true or false Nick s age: age Nick = 7, A (7) = (true) age Nick = 69.9, A (69.9) = (false) A Eindhoven, the Netherlands 5 6 Fuzzy set Fuzzy proposition Graded membership, element belongs to a set to a certain degree. membership grade A Nick is old... degree of truth age Nick = 7, A (7) =.5 age Nick = 69.9, A (69.9) =.49 age Nick = 9, A (9) = A membership grade Eindhoven, the Netherlands 7 8 3

4 Typical linguistic values Linguistic variable membership grade young middle age old x is age = {young, middle age, old} membership grade young middle age old semantic rules M X Eindhoven, the Netherlands 9 2 Fuzzy complement Intersection of fuzzy sets (x) = A (x) AB (x) = min( A (x), B (x)) A A B x x Eindhoven, the Netherlands 2 Eindhoven, the Netherlands 22 Union of fuzzy sets AB (x) = max( A (x), B (x)) A B FUZZY SYSTEMS x Eindhoven, the Netherlands 23 4

5 Linguistic variable Where: {x,,, M X } x name of the linguistic variable linguistic values (terms) Universe of discourse M X semantic rule that associates each linguistic value to a membership function. Fuzzy if-then rules Fuzzy propositions x is A, y is B Linguistic (Mamdani) fuzzy if-then rule: Antecedent: x is A Consequent: y is B If x is A then y is B Rule If x is A then y is B is represented by a fuzzy relation defined on X Y. Eindhoven, the Netherlands 25 Eindhoven, the Netherlands 26 Examples If the road is slippery then brake softly. If error is Negative big and e is Positive big then u is Negative small. If a tomato is red then the tomato is ripe. If the temperature is very high then reduce the heat a lot. If the valve is closed then the pressure is high. Linguistic (Mamdani) model k k k R : Ifxis A thenyis B, k,2,, K Decomposing using conjunctive forms: R : If x is A and x is A andand x is A k k k k 2 2 n n k k k then yis B and y2 is B2 andand yp is Bp Degree of fulfillment of antecedents: k = ( x ) ( x ) ( x ), k,2,, K A 2 k A2 k A k n n Eindhoven, the Netherlands 27 Eindhoven, the Netherlands 28 Takagi-Sugeno fuzzy model R k : If xis A k then y k f k ( x), k,2,, K Affine linear form: T k k k k k R : If xis A then y a xb Degree of fulfillment k defined as in linguistic models Model output given by the weighted fuzzy-mean: y K k k K k k T k y ( a ) xb k k K K j j j j Bibliography G. Klir and T. Folger. Fuzzy Sets Uncertainty and Information. Prentice Hall, 988. J.-S. Jang, C.-T. Sun and E. Mizutani. Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence. Prentice Hall, New Jersey, 997. Andries P. Engelbrecht. Computational Intelligence: An Introduction. John Wiley, Chichester, 22. J.M.C. Sousa and U. Kaymak. Fuzzy Decision Making in Modeling and Control. World Scientific Series in Robotics and Intelligent Systems, vol. 27. World Scientific Pub. Co., Singapore, Dec. 22 Michael Negnevitsky. Artificial Intelligence: A Guide to Intelligent Systems. Addison-Wesley, Pearson Education, 22. R. Babuska. Fuzzy Modeling for Control. Kluwer Academic Publishers, 998. Eindhoven, the Netherlands 29 Eindhoven, the Netherlands 3 5

6 Kernel-based modeling FUZZY MODELING Fuzzy systems Radial basis function networks Support vector machines Multi-layer perceptron... Fuzzy systems can be interpretable! Fuzzy sets can close the gap between symbolic processing and numerical computations. Eindhoven, the Netherlands 32 Fuzzy system parameters Parameters of antecedent membership functions (shape, location, etc.) Parameters of consequent membership functions (Mamdani systems) Parameters of consequent functions (Takagi-Sugeno systems) Aggregation of antecedent memberships Implication/reasoning Defuzzification function (Mamdani systems) Building fuzzy models Data-driven approach nonlinear mapping extract from input-output data: rules antecedents (membership functions) consequents (membership or crisp functions) Eindhoven, the Netherlands 33 Eindhoven, the Netherlands 34 Fuzzy c-means Modeling based on fuzzy clustering MF.5.5 Y Assumes partition matrix is fixed.5 X.5 MF MF.5.5 Y X.5. Collect the data 2. Select model structure (Mamdani, Takagi-Sugeno, ) 3. Select number of clusters and clustering algorithm 4. Cluster the data 5. Obtain antecedent membership functions (MF) from clusters. Obtain consequents (MF or parameters) 6. Determine a fuzzy rule for each cluster 7. Simplify the model, if necessary 8. Validate the model.5 Y X.5 35 Eindhoven, the Netherlands 36 6

7 Building fuzzy models Structure Input and output variables. For dynamic systems also the representation of the dynamics. Number of membership functions per variable, type of membership functions, number of rules. Parameters Antecedent membership functions Consequent parameters Linguistic models from clustering k k k R : Ifxis A thenyis B, k,2,, K Use fuzzy c-means algorithm. Cluster data in input output product space. Membership functions obtained by: projection onto variables, membership function parameterization. One rule per cluster Eindhoven, the Netherlands 37 Eindhoven, the Netherlands 38 Example of linguistic model Selecting number of antecedents If income is Low then tax is Low If income is High then tax is High A priori knowledge (experts, dynamics, etc.) Regularity criterion based on cross-validation. Split training set randomly into two parts (A and B) Minimize regularity criterion: RC y y y y ka kb A AB B BA ˆ ˆ i i k B i i 2 2 2kA i i Variables selected incrementally until regularity criterion increases. Figure reproduced with permission of Prof. Uzay Kaymak 39 Eindhoven, the Netherlands 4 Selecting number of antecedents Feature selection (FS) Principal Component Analysis (PCA) Curvilinear Component Analysis (CCA) (...) Tree search methods Bottom-up Top-down FS using genetic algorithms FS using ant colony optimization FEATURE SELECTION Eindhoven, the Netherlands 4 7

8 Introduction Many applications have hundreds to tens of thousands of variables/features Many are irrelevant and/or redundant. Curse of dimensionality. Feature selection What is feature selection? Remove features (inputs) X(i) to improve (or least degrade) prediction of outputs Y. Advantages: Feature selection selects most relevant features Collect/process less features and data Less complex models run faster Models are easier to understand, verify and explain 2 September 2 Eindhoven, the Netherlands 43 Eindhoven, the Netherlands 44 Feature selection algorithms Tree search bottom-up Filters Based on general characteristics of data to be evaluated. No model is involved. Wrappers Uses model performance to evaluate feature subsets. Train one classification model for each feature subset. Hybrid methods Do not retrain the model at every step. Search feature selection space and model parameter space simultaneously. Eindhoven, the Netherlands 45 2 September 2 Eindhoven, the Netherlands 46 Biologically inspired algorithms ANT COLONY OPTIMIZATION Artificial ant colonies: maybe the most used method from the artificial life algorithms. Introduced by Marco Dorigo (992), has been well received by academic world and it is starting to be used in industrial applications. Applications: Traveling Salesman Problem, Vehicle Routing, Quadratic Assignment Problem, Internet Routing, Logistic Scheduling, clustering and data mining problems. Eindhoven, the Netherlands 48 8

9 Ant Colony Optimization What is special about ants? Artificial Life algorithms: swarm, ants, wasps, bees Ant Colony Optimization is one of the most used method of the Artificial Life algorithms. Applications: Travelling salesman problem, vehicle routing, quadratic assignment problem, internet routing, logistics scheduling. There are also some applications of ACO in clustering and data mining problems, including feature selection. Ants can perform complex tasks: nest building, food storage garbage collection, war foraging (to wander in search of food) There is no management in an ant colony collective intelligence They communicate using: pheromones (chemical substance), sound, touch Curiosities: Ant colonies exist for more than million years Myrmercologists estimate that there are around 2 species of ants Eindhoven, the Netherlands 49 Eindhoven, the Netherlands 5 The foraging behaviour of ants Artificial ants How can almost blind animals manage to learn the shortest route paths from their nests to the food source and back? a) - Ants follow path between the Nest and the Food Source c) - On the shorter path, more pheromones are laid down b) - Ants go around the obstacle following one of two different paths with equal probability d) At the end, all ants follow the shortest path. Fotos: Eindhoven, the Netherlands 5 Artificial ants move in graphs nodes / arcs environment is discrete As real ants: choose paths based on pheromone concentration deposit pheromones on paths Environment updates pheromones Extra abilities of artificial ants: prior knowledge (heuristic ) memory (feasible neighbourhood N 52 Mathematical framework Bibliography Choose node ij ij, if j N k p ij ij ij j, otherwise Update feasible neighbourhood N N \ j Pheromone update k ( l ) ( l) ( ), if(, ) k Q f i j S ij, otherwise ij Initialization Set ij = For l =: N max Build a complete tour For i = to n For k = to m Choose node Update N Apply local heuristic end end Analyze solutions For k = to m Compute f k end Update pheromones end Marco Dorigo and Thomas Stützle. Ant Colony Optimization. The MIT Press. July 24. J. Kennedy, R. C. Eberhart and Y. Shi. Swarm Intelligence. Morgan Kaufmann Publishers, 22. Andries P. Engelbrecht. Computational Intelligence: An Introduction. John Wiley, Chichester, 22 Eindhoven, the Netherlands

10 Objective function ANT FEATURE SELECTION Objectives: minimize the number of misclassifications, or the classification error N e reduce the number of features, or the feature cardinality N f minimize f wne w2nf Tradeoff precision vs. accuracy. 2 September 2 Eindhoven, the Netherlands 56 Multicriteria ant system Ant Feature Selection (AFS) Feature Feature N Rank Features Update pheromone Ant colony for cardinality of features Ant system Update pheromone Ant colony for selection of features X test Test Modeling Y test Choose node ij ij, if j N k p ij ij ij j, otherwise x 3 x x2 x 4 Minimize number of features N cycles Cost Minimize classification error Pheromone update k ( l) ( l)( ) ij x 5 x 7 x n Subset: {x 3,x 6,x 7,x,x 4 } x 6 2 September 2 Eindhoven, the Netherlands 57 2 September 2 Eindhoven, the Netherlands 58 Heuristics in AFS Heuristic for feature cardinality: Fisher s score for the features () i () i Fi () () () i c c2 2 2 c i c2 2 mean and variance values of feature i for the samples in class c and c 2 Heuristic for selection of features: classification error e(i) for the individual features f() i ei () Ant feature selection General design Number of ants g Balance of exploration and exploitation Combination with greedy heuristics or local search When should pheromones be updated? 2 September 2 Eindhoven, the Netherlands 59 Eindhoven, the Netherlands 6

11 Algorithm /* Initialization */ Set parameters f, n, f, n, f, n, I, N, g. for t = to I Choose size of subset N f (k) for each ant k for l = to N Build feature set L k f(t) choosing N f (k) features Derive model using L k f(t) features selected by ant k Compute classification error E k (t) Update pheromone trails ni (t + ) and fj (t + ) end for end for Eindhoven, the Netherlands 6 PREDICTING OUTCOMES OF SEPTIC SHOCK PATIENTS Motivation Problem Septic shock is a common ICU key adverse outcome, translated into ~5% mortality rate and high costs of treatments. Goal Predict the outcome (survive or decease) of septic shock patients, for purposes of therapy management. Methods Fuzzy Systems or Neural Networks + Feature Selection (tree search and ant colony optimization) Sepsis Annual mortality rate of sepsis in USA: more than 22,. Sepsis is the tenth most common cause of death. Severe sepsis accounts 2% to 3% of all hospital admissions. 59% of patients with sepsis require ICU care, composing.4% of ICU admissions. The mortality rate for severe sepsis ranges from 3% to 5%, and is as high as 8% to 9% for septic shock and multiple organ dysfunction. 2 September 2 Eindhoven, the Netherlands 63 2 September 2 Eindhoven, the Netherlands 64 Septic shock - background Management of sepsis is increasingly protocol-driven Care is goal-directed and parameterized With goal-directed therapy, care becomes similar to a control problem, with ideal process of care revolving around: Setting a goal/target for a specific physiological parameter Rapidly driving the physiologic process toward specific goal/target Maintaining that physiological parameter within upper and lower limits of that goal Septic shock - assumptions Adequacy of control depends largely on: Close monitoring Early detection of change Active management and intervention by nurses 2 September 2 Eindhoven, the Netherlands 65 2 September 2 Eindhoven, the Netherlands 66

12 MEDAN database Problems in the database Database used as testbench (Paetza 23) Variables The MEDAN data base contains the data of 3 variables of 387 patients Data from ICU from collected by medical documentation staff All patients have septic shock of abdominal cause Task Predict patients survival Selection of 387 patients and 59 variables. 2 September 2 Eindhoven, the Netherlands 67 2 September 2 Eindhoven, the Netherlands 68 Problems in the database One of the most complete patients. Problems in the database Measurements for a considerable part of the variables stopped. Variable Variable Time [hours] Time [hours] 2 September 2 Eindhoven, the Netherlands 69 2 September 2 Eindhoven, the Netherlands 7 Problems in the database Long periods with missing data. Variable Classification measures In this example we used the following measures: Classification accuracy (% of correct classification) Area under the ROC Curve (AUC) Specificity Sensitivity Time [hours] 2 September 2 Eindhoven, the Netherlands 7 2 September 2 Eindhoven, the Netherlands 72 2

13 Confusion matrix Specificity and Sensibility Specificity or true negative rate (TNR) TN Specificity TN FP Sensitivity or true positive rate (TPR) TP Sensibility TP FN 2 September 2 Eindhoven, the Netherlands 73 2 September 2 Eindhoven, the Netherlands 74 Area Under the ROC Curve (AUC) In signal detection theory, a receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot of the sensitivity, vs. false positive ratio ( specificity). Area under the ROC curve (AUC) Results Classification accuracy ACC (%) FS method - Tree search AFS Model NN [Paetza] 2 Features set 28 Features set Fuzzy NN Fuzzy NN September 2 Eindhoven, the Netherlands 75 2 September 2 Eindhoven, the Netherlands 76 Results Specificity Results Sensitivity FS method - Tree search AFS Model NN [Paetza] 2 Features set 28 Features set Fuzzy NN Fuzzy NN FS method - Tree search AFS Model NN [Paetza] 2 Features set 28 Features set Fuzzy NN Fuzzy NN September 2 Eindhoven, the Netherlands 77 2 September 2 Eindhoven, the Netherlands 78 3

14 Results AUC 2 features subset % 8% FS method - Tree search Model NN [Paetza] 2 Features set 28 Features set Fuzzy NN Frequency 6% 4% 2% % Feature label BU + FM AFS + FM AFS + NN BU + NN AFS Fuzzy NN Most frequent features: 8 ph 26 Calcium 28 Creatinine 2 September 2 Eindhoven, the Netherlands 79 2 September 2 Eindhoven, the Netherlands 8 28 features subset Future work Frequency % 9% 8% 7% 6% 5% 4% 3% 2% % % Feature label Most frequent features: 8, 26, 28 and 8 thrombocytes 4 CRP (C-reactive protein) 22 antithrombiniii 85 FiO2 35 total bilirubin BU + FM AFS + FM AFS + NN BU + NN Apply the same techniques to larger health care databases with more available features (MIMIC II) MIMIC II (dimension of database) 4, patients and 5 features Preprocessing Large amount of missing values Uneven time samplings Validate models with other datasets Hospital da Luz in Lisbon. 2 September 2 Eindhoven, the Netherlands 8 2 September 2 Eindhoven, the Netherlands 82 4