ACO, NATURAL AGENTS APPLIED TO FEATURE SELECTION

Size: px

Start display at page:

Download "ACO, NATURAL AGENTS APPLIED TO FEATURE SELECTION"

Clarence Reeves
5 years ago
Views:

1 Multi agent systems ACO, NATURAL AGENTS APPLIED TO FEATURE SELECTION Susana Vieira Technical University of Lisbon, Instituto Superior Técnico Dept. of Mechanical Engineering, Center of Intelligent Systems/IDMEC Av. Rovisco Pais 1, Lisboa, Portugal E mail: susana@dem.ist.utl.pt Agents basic characteristics 1 : Autonomy: the agents are at least partially autonomous; Local views: no agent has a full global view of the system, or the system is too complex for an agent to mae practical use of such nowledge; Decentralization: there is no designated controlling agent. 1 Michael Wooldridge, An Introduction to MultiAgent Systems, John Wiley & Sons, 2002, ISBN X. Susana Vieira November 4 th, What happens in nature? Outline "An individual ant is not very bright, but ants in a colony, operating as a collective, do remarable things. A single neuron in the human brain can respond only to what the neurons connected to it are doing, but all of them together can be Albert Einstein." By Deborah M. Gordon (Stanford University) We are interested in systems where simple units together behave in complicated ways Swarm Intelligence Ant colony optimization Feature selection Ant feature selection Examples Conclusions and future wor Susana Vieira November 4 th, Susana Vieira November 4 th,

Learning from Nature Nature has inspired researchers in many different ways. Airplanes have been designed based on the structures of birds' wings.

After millions of years of eoltionall evolution these species developed solutions for a wide range of problems. Some ideas can be developed by taing advantage of the examples that Nature offers.

The intelligent solutions to problems naturally emerge from the self organization and communication of these individuals.

Susana Vieira November 4 th, 2009 5 Susana Vieira November 4 th, 2009 6 Swarm Intelligence Based on the study of emergent collective intelligence of groups of simple agents Bird Floc Swarm

2 Learning from Nature Nature has inspired researchers in many different ways. Airplanes have been designed based on the structures of birds' wings. Robots have been designed in order to imitate the movements of insects. Resistant materials have been synthesized based on spider webs. After millions of years of eoltionall evolution these species developed solutions for a wide range of problems. Some ideas can be developed by taing advantage of the examples that Nature offers. Learning from Nature Some social systems in Nature can present an intelligent collective behavior although they are composed by simple individuals. The intelligent solutions to problems naturally emerge from the self organization and communication of these individuals. These systems provide important techniques that can be used in the development of distributed artificial intelligent systems. Susana Vieira November 4 th, Susana Vieira November 4 th, Swarm Intelligence Based on the study of emergent collective intelligence of groups of simple agents Bird Floc Swarm Intelligence Swarm Intelligence is an artificial intelligence technique based on the study of collective behavior in selforganized systems. Swarm Intelligence systems are typically made up of a population of simple agents interacting locally with one another and with their environment. This interaction often lead to the emergence of global behavior. Ant Colony Animal Herd The main bio inspired algorithms that have been developed are: Ant Colony Optimisation (ACO) Particle Swarm Optimisation (PSO) Fish School Susana Vieira November 4 th, Susana Vieira November 4 th,

3 Natural ants Individual ants are simple insects with limited memory and capable of performing simple actions. However, an ant colony expresses a complex collective behavior providing intelligent solutions to problems such as: carrying large items forming bridges finding the shortest routes from the nest to a food source, prioritizing food sources based on their distance and ease of access. What is special about ants? Ants can perform complex tass: nest building, food storage garbage collection, war foraging (to wander in search of food) There is no management in an ant colony collective intelligence They communicate using: pheromones (chemical substances), sound, touch Curiosities: Ant colonies exist for more than 100 million years Myrmercologists estimate that there are around species of ants Susana Vieira November 4 th, Susana Vieira November 4 th, Ant colony optimization The foraging behaviour of ants Ant Colony Optimization is one of the most used method of the Artificial Life algorithms. Introduced by: Marco Dorigo (1992), and is starting to be used in industrial applications. Applications: Travelling salesman problem, vehicle routing, quadratic assignment problem, internet routing, logistics scheduling. There are also some applications of ACO in clustering and data mining problems, including feature selection. Susana Vieira November 4 th, How can almost blind animals manage to learn the shortest route paths from their nests to the food source and bac? a) Ants follow path between the Nest and the Food Source c) On the shorter path, more pheromones are laid down b) Ants go around the obstacle following one of two different paths with equal probability d) At the end, all ants follow the shortest path. Fotos: Susana Vieira November 4 th,

Mathematical framewor Artificial ants Choose trail i = 1 τ 12 Deposit pheromone τ 13 j = 2 Environment (time) updates pheromones Time is the performance index ρ is the evaporation coefficient i = 1 j

ants: choose paths based on pheromone concentration deposit pheromones on paths environment updates pheromones Extra abilities of artificial ants: prior nowledge (heuristic η) memory (feasible

4 Mathematical framewor Artificial ants Choose trail i = 1 τ 12 Deposit pheromone τ 13 j = 2 Environment (time) updates pheromones Time is the performance index ρ is the evaporation coefficient i = 1 j = 3 Ant 1, t=0 i = 1 τ 12 τ 12 p = f( τ ) j = 2 τ 13 + Δτ τ 13 j = 2 Ant 2, t=2 Ant 1, t=1 j = 3 τ ( t+ n) =τ ( t) ρ+δτ Artificial ants move in graphs nodes / arcs environment is discrete As real ants: choose paths based on pheromone concentration deposit pheromones on paths environment updates pheromones Extra abilities of artificial ants: prior nowledge (heuristic η) memory (feasible neighbourhood N) Food Source Destination Nest Source Susana Vieira November 4 th, Susana Vieira November 4 th, Mathematical framewor Initialization Set τ = τ 0 For l =1: N α β max τ η Builda complete tour, if j N For i = 1 to n = τ α η β For = 1 to m j Ν Choose node 0, otherwise Update N Apply local heuristic end end Analyze solutions l+ = l +Δ For = 1 to m Compute f end Update pheromones end Choose node p Pheromone update τ( 1) τ()(1 ρ) τ ACO IN FEATURE SELECTION Susana Vieira November 4 th,

5 Motivation Feature selection Knowledge discovery process: Interpretation Feature selection almost always improves model accuracy Data acquisition Data Preprocessing Target data Feature selection Preprocessed data Modeling Reduced data Patterns Knowledge Benefits: Feature selection chooses the most relevant features Collect/process less features Less complex models run faster and are easier to understand, verify and explain Based on G. Piatetsy Shapiro U. Fayyad and P. Smyth. From data mining to nowledge discovery in databases. Artificial Intelligence Magazine, 17(3):37 54, Susana Vieira November 4 th, Susana Vieira November 4 th, Feature selection What is feature selection? Remove features X(i) to improve (or least degrade) prediction of Y. Objectives: reduce model complexity and computationalloadwithout load loosingaccuracy Feature selection algorithms Filters Based on general characteristics of data to be evaluated. No model is involved. Wrappers Uses model performance to evaluate feature subsets. Train one classification model for each feature subset. Hybrid methods Do not retrain the model at every step. Search feature selection space and model parameter space simultaneously. Susana Vieira November 4 th, Susana Vieira November 4 th,

6 Objective function Optimization algorithm Main objectives: Multicriteria algorithm: Minimize thenumber ofmisclassifications misclassifications, or the classification error Reduce the number of features, or the features cardinality Feature 1 Feature N Ran Features Update pheromone Ant colony for cardinality of features Ant system Update pheromone Ant colony for selection of features X test Test Modeling Y test minimize f = w e + w N 1 2 Tradeoff precision vs. accuracy. Minimize number of features N cycles Cost Minimize classification error Susana Vieira November 4 th, Susana Vieira November 4 th, ACO for feature selection Choosing node in graph Second colony: Choose node x 1 x2 x 2 α β τ η, if j N x 3 α β p = τ η j Ν 0, otherwise Pheromone update τ( l+ 1) = τ()(1 l ρ) +Δτ x 5 x 7 x n Subset: {x 3,x n,x 7,x 1,x 4 } x 4 x 6 Probability of an ant choosing node i (cardinality of features): Probability of an ant choosing node j (selection of features): α f β f τ f () t η j fj pj () t = α f β f τ f () t η jl fjl α β n n τ n() t η i ni pi () t = αn τn () t η il nil l Ji l J j β n Susana Vieira November 4 th, Susana Vieira November 4 th,

7 Heuristics of ant systems Modeling Heuristic for feature cardinality: Fisher s score for the features Fi () = μ () i μ () i σ () () i c1 c2 2 2 c i + σ 1 c2 2 mean and variance values of feature i for the samples in class c 1 and c 2 Heuristic for selection of features: classification error e(i) ( ) for the individual features 1 η f () i = ei () Taagi Sugeno fuzzy models are used; Antecedents A i are fuzzy sets obtained using fuzzy clustering membership functions. Consequents y i are estimated using least squares estimation. Feedforward Neural Networ are used; Susana Vieira November 4 th, Susana Vieira November 4 th, Data sets Results (Wine) Examples: Data sets Number of features Number of classes Size of data set Wine Breast cancer Reduced Classification accuracy (%) Subsets Best Mean Worst AFS Approach Corcoran and Sen (1994) Ishibuchi et al. (1999) Roubos et al. (2003) Mendonça et al. (T D) (2007) Mendonça et al. (B U) (2007) Susana Vieira November 4 th, Susana Vieira November 4 th,

8 Results (Breast Cancer) Fuzzy goals and constraints Reduced Subsets Classification accuracy (%) Best Mean Worst AFS Approach Wang et al. (POSAR) (2004) Wang et al. (CEAR) (2004) Wang et al. (DISMAR) (2003) Wang et al. (GAAR) (2000) Wang et al. (PSORSFS) (2007) Abony et al. (GG: R = 2) (2003) Abony et al. (Sup: R = 2) (2003) Abony et al. (GG: R = 4) (2003) Abony et al. (Sup: R = 4) (2003) Let A be a given set of possible alternatives which contains asolution to a decision maing problem under consideration. i A fuzzy goal G is a fuzzy set on A, characterized by μ G : A [0,1], represents the degree to which the alternatives satisfy the specified decision goal. A fuzzy constraint C is a fuzzy set on A characterized by μ C : A [0,1], constrains the solution to a fuzzy region within the set of possible solutions.. Susana Vieira November 4 th, Susana Vieira November 4 th, Fuzzy goal Fuzzy constraint Goal: Product concentration should be about 80%. mem mbership grade About 80 % Constraint: Product concentration should be not substantially higher than 75%. mem mbership grade Not substantially higher than 75 % Susana Vieira November 4 th, Susana Vieira November 4 th,

9 Bellman and Zadeh s model Optimal fuzzy decision Fuzzy decision F is a confluence of (fuzzy) decision goals and (fuzzy) decision criteria Both the decision i goals and the decision i constraints should ldbe satisfied μ μ μ F = G C ( a) = ( a) ( a), a A F G C Maximising decision (optimal decision a*) Decision with the largest membership value a* = arg max ( a) ( a) a A μ μ G Alternative corresponding to the largest membership value is denoted as the best alternative (solution) Susana Vieira November 4 th, C Maximizing decision using min: ershipgrade membe x m x m = μ D* Fuzzy Decision μ D Susana Vieira November 4 th, BZ model : example Fuzzy criteria in feature selection μ Small dosage (fuzzy constraint) 1 fuzzy decision Large dosage (fuzzy goal) Two criteria are considered: classification error F 1 rship Grade 1 μ e maximizing decision interferon dosage [mg] Membe 0 0 K e e Number of errors Susana Vieira November 4 th, Susana Vieira November 4 th,

10 Fuzzy criteria in feature selection features cardinality F 2 1 μ nf Memb bership Grade 0 0 K K nf Nfmax 100 Number of features nf Fuzzy optimization Fuzzy goal F j, j = 1,2,...,n Membership functions: F j (x): X [0,1] j Fuzzy decision (Bellman and Zadeh model): D(x) = F 1 (x)... F n (x) Optimal decision: x * = arg max Dx ( ) x X Susana Vieira November 4 th, Susana Vieira November 4 th, Fuzzy objective function Fuzzy criteria results Classic objective function minimize f = w e+ w N 1 2 f Data set Reduced Subsets Classification accuracy (%) Best Mean Worst Fuzzy objective function D(x) = F 1 (x)... F 2 (x) maximize D() x ( ) D() x = I( F, w ),( I F, w ) Wine AFS (test) Fuzzy AFS Breast cancer AFS (cross validation) Fuzzy AFS Susana Vieira November 4 th, Susana Vieira November 4 th,

11 Results: classical vs fuzzy criteria Data sets bjective function O Classical versus fuzzy objective function convergence: Wine Classical objective function Fuzzy objective function Iteration number bjective function O Breast Cancer Classical objective function Fuzzy objective function Iteration number Examples: Data sets Number of Number of Size of features classes data set 1 Breast cancer original Wine Vote Diagnostic breast cancer Prognostic breast cancer Sonar Mus Susana Vieira November 4 th, Susana Vieira November 4 th, Results: classical vs fuzzy criteria Classification rates with 10 fold cross validation: data FuzzyModels Neural Networs set No FS AFS NF FOF NF No FS AFS NF FOF NF Ave WTL 0/0/7 0/1/6 6/1/0 0/0/7 0/1/6 6/1/0 Conclusions Ants are natural multi agents systems. A feature selection algorithm based on two cooperative ant colonies was presented. The problem is divided into two contradictory objectives: choosing the features cardinality and selecting the most relevant features. Fuzzy objective functions for featureselection are used. Fuzzy objective functions help the convergence of ACO. Susana Vieira November 4 th, Susana Vieira November 4 th,

12 Future wor Than you all! New measures are being used in Ant feature selection (AFS): Cohen s appa coefficient. Application problems: Systems redesign to improve the survival of critically ill patients using data based modeling Two problems in ICU are considered: sepsis and selfextubation. Problems addressed: number of features, missing data, outliers. Prof. Uzay Kayma Econometric Institute, Erasmus School of Economics Erasmus University of Rotterdam Prof. João M. C. Sousa Center of Intelligent Systems IDMEC Instituto Superior Técnico Technical University of Lisbon Susana Vieira November 4 th, Susana Vieira November 4 th, Data sets Results (Wine) Examples: Data sets Number of Number of Size of features classes data set Wine Breast cancer Vote M_of_N Sonar Reduced Classification accuracy (%) Subsets Best Mean Worst AFS Approach Corcoran and Sen (1994) Ishibuchi et al. (1999) Roubos et al. (2003) Mendonça et al. (T D) (2007) Mendonça et al. (B U) (2007) Susana Vieira November 4 th, Susana Vieira November 4 th,

13 Results (Wine) Results (Breast Cancer) Minimum number of errors and best number of features for each iteration in Wine data set: Reduced Subsets Classification accuracy (%) Best Mean Worst AFS Approach Wang et al. (POSAR) (2004) Wang et al. (CEAR) (2004) Wang et al. (DISMAR) (2003) Wang et al. (GAAR) (2000) Wang et al. (PSORSFS) (2007) Abony et al. (GG: R = 2) (2003) Abony et al. (Sup: R = 2) (2003) Abony et al. (GG: R = 4) (2003) Abony et al. (Sup: R = 4) (2003) Susana Vieira November 4 th, Susana Vieira November 4 th, Results (Breast Cancer) Results (Vote) Minimum number of errors and best number of features for each iteration in Breast Cancer data set: Reduced Classification accuracy (%) Subsets Best Mean Worst AFS Approach Wang et al. (POSAR) (2004) Wang et al. (CEAR) (2004) Wang et al. (DISMAR) (2003) Wang et al. (GAAR) (2000) Wang et al. (PSORSFS) (2007) Susana Vieira November 4 th, Susana Vieira November 4 th,

14 Results (Vote) Results (M of N) sification accuracy Minimum clas Minimum number of errors and best number of features for each iteration in Vote data set: iteration ber of features Best Numb iteration Reduced Classification accuracy (%) Subsets Best Mean Worst AFS Approach Wang et al. (POSAR) (2004) Wang et al. (CEAR) (2004) Wang et al. (DISMAR) (2003) Wang et al. (GAAR) (2000) Wang et al. (PSORSFS) (2007) Susana Vieira November 4 th, Susana Vieira November 4 th, Results (M of N) Minimum number of errors and best number of features for each iteration in M of N data set: Results (Sonar) Comparison results: classification accuracy Average c # iteration (I) umber of features Best Nu # iteration Number Reduced Test accuracy (%) of rules Subsets Average best error rate (%) AFS approach Ishibuchi et al. (2007) 10 all 82.7 Susana Vieira November 4 th, Susana Vieira November 4 th,

15 Best Number of Features Results (Sonar) Minimum number of errors and best number of features for each iteration in Sonar data set: # Iteration (I) Minimum m classification accuracy # Iteration (I) Results Computational time and number of rules: Dataset #Samples #Features #Rules Time (s) Wine Breast Cancer Vote M of N Sonar Susana Vieira November 4 th, Susana Vieira November 4 th,

Feature Selection in Knowledge Discovery

Feature Selection in Knowledge Discovery Susana Vieira Technical University of Lisbon, Instituto Superior Técnico Department of Mechanical Engineering, Center of Intelligent Systems, IDMEC-LAETA Av. Rovisco