LECTURE 20: SWARM INTELLIGENCE 6 / ANT COLONY OPTIMIZATION 2

Transcription

1 COLLECTIVE INTELLIGENCE - S18 LECTURE 20: SWARM INTELLIGENCE 6 / ANT COLONY OPTIMIZATION 2 INSTRUCTOR: GIANNI A. DI CARO

2 ANT-ROUTING TABLE: COMBINING PHEROMONE AND HEURISTIC 2

3 STATE-TRANSITION: MOVING TO THE NEXT HOP/ STATE Source τ 12;η 12 1 τ 13;η 13 τ ;η τ ;η τ ;η Destination Pheromone Intensity Scale 6 3

4 FUNCTIONAL FORM OF THE STOCHASTIC DECISION POLICY 4

5 POSSIBLE STRATEGIES FOR PHEROMONE UPDATING 5

6 ANT AGENT: THE BEHAVIOR 6

7 ANT SYSTEM (1994) 7

8 ANT SYSTEM (1994) 8

9 AS: EXPLORATION - EXPLOITATION TRADEOFF 9

10 AS: PHEROMONE EVAPORATION 10

11 AS: PHEROMONE UPDATE Pheromone is iteratively deposited in an additive cumulative modality based on solution quality 11

12 QUESTIONS 1. Why an additive, cumulative rule for pheromone updating and not an average, for instance? (not looking for averages, but for the sparse best solutions) 2. Is there any potential problem with pheromone bounds? (get to zero, unlimited growth) 3. Is there any potential problem of premature convergence? 4. Is it a good idea to have a large number of samples / ants given the adopted rule for pheromone updating? (all solutions do pheromone updating A lot of bad ones!) 5. How do we balance policy evaluation and policy improvement? 12

13 AS: OTHER PHEROMONE UPDATE RULES Idea: assign credits relative to some Q costant value related to problem s costs Q = an upper bound estimate on the length of the optimal tour, in Ant-cycle Q = small value related to the range of cost values, Ant-density & Ant-Quantity 13

14 AS: ELITIST PHEROMONE UPDATE 14

15 ANT COLONY SYSTEM (1998) ACS addresses main AS shortcomings and introduces new components A different transition rule is used A different pheromone update rule is defined Step-by-step local pheromone updates are introduced Candidate lists are used to favor specific nodes and save a lot of computation (at each step, check among n E possible decisions, E can easily be 10 k, k > 3) Later (and more performing) versions make use of a daemon component based on local search SoA heuristic for TSP and similar problems 15

16 ANT COLONY SYSTEM (1996) Function AntColonySystem() Pheromone model: ij (estimated) quality of selecting city j when i is the current city Heuristic variables: ij 1/distancefromcityi to city j m number of ants per iteration (i.e., samples for policy evaluation) nn tour find initial solution with nearest neighbor heuristic Init pheromone: ij = 0 =1/(n nn tour ), 8i, j 2 {1,...,n} for t := 1,... iterations num in-parallel for k := 1,...,m /* Ants construct solutions in parallel */ T k (t),l k (t) ant construct solution(online step by step pheromone update) /* Only best ant tour generated so far is selected for pheromone update */ best so far ant update pheromone({t k (l),l k (l)},l =1,...,t, k =1,...,m) return best solution generated 16

17 ACS: TRANSITION RULE ɛ-greedy policy 17

18 ACS: EXPLOITATION - EXPLORATION TRADEOFF 18

19 ACS: PHEROMONE UPDATE AND EVAPORATION We are looking for the best, not the average 19

20 ACS: PHEROMONE UPDATE AND EVAPORATION Persistence, conservative approach: For small values of ρ1, the existing pheromone concentrations on the edges evaporate slowly, while the influence of the best route is dampened Volatile, aggressive approach: For large values of ρ1, previous pheromone deposits evaporate rapidly, but the influence of the best path is emphasized The effect of large ρ1 is that previous experience is neglected in favor of more recent experiences more exploration Simulated Annealing approach: If ρ1 is adjusted dynamically from large to small values, exploration is favored in the initial iterations of the search, while focusing on exploiting the best found paths in the later iterations 20

21 ACS: ONLINE PHEROMONE UPDATE A good choice is potentially made locally less good after being selected. This is to favor exploring other local choices during the same iteration loop Pheromones don t go to zero! 21

22 ACS: CANDIDATE LISTS 22

23 ACS: (OLD) PERFORMANCE (1997) TSP problems from TSPLIB Euclidean TSP instances GA = Genetic algorithm EP = Evolutionary programming SA = Simulated annealing Table shows the best integer tour length, the best real tour length (in parentheses), and the number of tours required to find the best integer tour length (in square brackets) Results out of 25 trials 23

24 ACS: (OLD) PERFORMANCE (1997) TSP problems from TSPLIB 24

25 ACS: DAEMON ACTION, LOCAL SEARCH Function AntColonySystem-3-Opt() Pheromone model: ij (estimated) quality of selecting city j when i is the current city Heuristic variables: ij 1/distancefromcityi to city j m number of ants per iteration (i.e., samples for policy evaluation) nn tour find initial solution with nearest neighbor heuristic Init pheromone: ij = 0 =1/(n nn tour ), 8i, j 2 {1,...,n} for t := 1,... iterations num in-parallel for k := 1,...,m /* Ants construct solutions in parallel */ T k (t),l k (t) ant construct solution(online step by step pheromone update) foreach T k (t), k:= 1,...,m /* Each ant solution becomes a starting point for 3-opt local search */ T k (t),l k (t) run 3-opt local search starting from ant tour(t k (t)) /* Only best ant tour generated so far is selected for pheromone update */ best so far ant update pheromone({t k (l),l k (l)},l =1,...,t, k =1,...,m) return best solution generated 25

26 ACS: DAEMON ACTION, LOCAL SEARCH At the end of each iteration, a local search is applied to all tours built by ants The resulting iteration (or global so far) best tour gets pheromone updating Selected LS: 3-Opt Computationally expensive, but rewarding! Symmetric TSP instances from the First International Contest on Evolutionary Optimization, IEEE-EC, May 20 22, 1996, Nagoya, Japan STSP GA is a GA with a Lin-Kernighan local search (called right after the crossover operator in order to produced a population of locally optimized individuals) 26

27 ACS: DAEMON ACTION, LOCAL SEARCH Asymmetric TSP instances (more difficult!) from the First International Contest on Evolutionary Optimization, IEEE-EC, May 20 22, 1996, Nagoya, Japan ATSP-GA is a genetic algorithm + local search 27

28 2-OPT LOCAL SEARCH 28

29 3-OPT LOCAL SEARCH 29

30 PHEROMONE AND HEURISTIC ARE BOTH IMPORTANT! 30

31 ANTS + LOCAL SEARCH As a matter of fact, the best instances of ACO algorithms for (static/centralized) combinatorial problems are those making use of a problem-specific local search daemon procedure (iterative solution modification) interleaved with ants solution construction It is conjectured that ACO s ants can provide good starting points for local search. More in general, a construction heuristic can be used to quickly build up a complete solution of good quality, and then a solution modification procedure can take this solution as a starting point, trying to further improve it by iteratively modifying some of its parts This hybrid two-phases search can be iterated and can be very effective if each phase can produce a solution which is locally optimal within a different class of feasible solutions, with the intersection between the two classes being minimal 31

32 MAX-MIN-AS (1999): LIMITS AND RESTARTS 32

33 MMAS (1999): LIMITS AND RESTARTS n 33

34 AS RANK (1999): ELITISM BY RANK 34

35 ANT-TABU (2001) 35

36 DYNAMIC, DISTRIBUTED ENVIRONMENTS? Routing in wired networks, AntNet (1998) Routing in mobile ad hoc networks (AntHocNet, 2005) Routing / Foraging in mobile robotic networks Challenges: Distributed, path evaluation, non-stationary, errors, unpredictable traffic demands, interference of ants with normal traffic, limited bandwidth, when/where send ants, where to store pheromone? 36

37 DECISIONS TO TAKE DESIGNING ACO ALGORITHMS 37

38 A FEW COMMON DESIGN CHOICES 38

39 THEORETICAL RESULTS? 39

40 ACO SUMMARY Reverse engineering of stigmergic pheromone laying-following in ant colonies Construction meta-heuristic biased by pheromone + heuristic Ants: Monte Carlo sampling of solutions Generalized policy iteration for learning pheromone parameters for decision-making: policy evaluation (sampling of solutions) + policy improvement (pheromone updating) Different strategies for sampling and updating A number of different heuristic recipes (common in SI, heuristic optimization domains) In physically distributed problems (e.g., networks, robotics) agents hops from one decision node to another and then have to retrace their path back, if feasible State of the art performance (when coupled with LS, for centralized problems) Guaranteed performance: yes, in the probabilistic limit Applied to a large variety of CO problems Large number of scientific publications Applied in the real world: Barilla, Migros, port management, logistics,. 40