An Overview of Search Algorithms With a Focus in Simulated Annealing

Transcription

1 An Overview of Search Algorithms With a Focus in Simulated Annealing K Jones Appalachian State University joneskp1@appstate.edu May 7, 2014

2 Definition of Annealing Definition: Annealing, in metallurgy and materials science, is a heat treatment that alters a material to increase its ductility and to make it more workable. It involves heating a material to above its critical temperature, maintaining a suitable temperature, and then cooling. [

3 The Problem Use the figure below to determine who the tallest person is.

4 The Solution Use the figure below to determine who the tallest person is.

5 The Second Problem Use the figure below to determine who the shortest person is.

6 Seurat s A Sunday on La Grande Jatte

7 The Real Problem

8 The Real Problem Use the data below to determine what the smallest value of Y is.

9 Solution Options Search Algorithms

10 Solution Options Select (Sort) Search Search Algorithms

11 Solution Options Search Algorithms Select (Sort) Search Hill Climb

12 Solution Options Search Algorithms Select (Sort) Search Hill Climb Simulated Annealing

13 Solution Options Search Algorithms Select (Sort) Search Hill Climb Simulated Annealing Tabu Search

14 Search Algorithms - Select (Sort) Search

15 Search Algorithms - Select (Sort) Search

16 Search Algorithms - Select (Sort) Search Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let y min y 0 and i 1 3: If y i < y min then y min y i 4: Let i i + 1 5: Repeat steps 3 and 4 while i n 1 6: Return y min

17 Search Algorithms - Select (Sort) Search Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let y min y 0 and i 1 3: If y i < y min then y min y i 4: Let i i + 1 5: Repeat steps 3 and 4 while i n 1 6: Return y min Upside: Guaranteed to find the best solution (i.e. optimal)

18 Search Algorithms - Select (Sort) Search Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let y min y 0 and i 1 3: If y i < y min then y min y i 4: Let i i + 1 5: Repeat steps 3 and 4 while i n 1 6: Return y min Upside: Guaranteed to find the best solution (i.e. optimal) Downise: There are n 1 comparisons to make which means this is a very expensive algorithm

19 Search Algorithms - Hill Climb

20 Search Algorithms - Hill Climb Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let i 0 and k 1 3: If y i < y k then return y i Else let i i + 1 and k k + 1 4: Repeat step 3 until k = n 1 5: Return y k if needed

21 Search Algorithms - Hill Climb Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let i 0 and k 1 3: If y i < y k then return y i Else let i i + 1 and k k + 1 4: Repeat step 3 until k = n 1 5: Return y k if needed Upside: Faster than Select Search since it doesn t always have to check the whole data list

22 Search Algorithms - Hill Climb Heuristic: 1: Let y i be the i th entry in an array of size n 2: Let i 0 and k 1 3: If y i < y k then return y i Else let i i + 1 and k k + 1 4: Repeat step 3 until k = n 1 5: Return y k if needed Upside: Faster than Select Search since it doesn t always have to check the whole data list Downise: Not guaranteed to find the optimal solution. Note: You will find an optimal solution

23 Search Algorithms - Simulated Annealing Definition: Simulated annealing (SA) is a generic probabilistic metaheuristic for the global optimization problem of locating a good approximation to the global optimum of a given function in a large search space. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For certain problems, simulated annealing may be more efficient than exhaustive enumeration provided that the goal is merely to find an acceptably good solution in a fixed amount of time, rather than the best possible solution. [

24 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized

25 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c )

26 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z +

27 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n )

28 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n ) 3: If y n < y b Then x b x n and y b y n

29 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n ) 3: If y n < y b Then x b x n and y b y n 4: If P(y c, y n, t) > Rand(0, 1) Then x c x n and y c y n

30 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n ) 3: If y n < y b Then x b x n and y b y n 4: If P(y c, y n, t) > Rand(0, 1) Then x c x n and y c y n 5: Let lcv lcv + 1

31 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n ) 3: If y n < y b Then x b x n and y b y n 4: If P(y c, y n, t) > Rand(0, 1) Then x c x n and y c y n 5: Let lcv lcv + 1 6: Repeat steps 2 through 5 until lcv = stop

32 Simulated Annealing - Heuristic Given: Domain X and function Y (x) which is to be minimized 1: Let x c, x b Rand(X ) and y c, y b Y (x c ) lcv 0, stop Z + 2: Let t Temp(lcv, stop), x n Neighbor(x c, t) and y n Y (x n ) 3: If y n < y b Then x b x n and y b y n 4: If P(y c, y n, t) > Rand(0, 1) Then x c x n and y c y n 5: Let lcv lcv + 1 6: Repeat steps 2 through 5 until lcv = stop 7: Return x b, y b

33 Simulated Annealing - Sub Routines

34 Simulated Annealing - Sub Routines Temp: In practice, a function is designed in terms of the proportion of loop time used. It is tailored for the specific optimization problem and computational limitations.

35 Simulated Annealing - Sub Routines Temp: In practice, a function is designed in terms of the proportion of loop time used. It is tailored for the specific optimization problem and computational limitations. 0 < Temp (lcv + 1, stop) < Temp (lcv, stop)

36 Simulated Annealing - Sub Routines Temp: In practice, a function is designed in terms of the proportion of loop time used. It is tailored for the specific optimization problem and computational limitations. 0 < Temp (lcv + 1, stop) < Temp (lcv, stop) Neighbor: The neighbor function randomly selects an element of X that is within some distance from x c. The distance from x c (i.e. width of the neighborhood) may remain constant or vary in realtionship to temperature. The function is chosen with the specific optimization problem and computational limitations in mind.

37 Simulated Annealing - Sub Routines P: The probability function is most commonly solved for using The Metropolis-Hastings Algorithm { e (yc yn)/t y c y n P(y c, y n, t) = 1 y c > y n The graphs below show the relationship of P in terms of D = y n y c with fixed temperatures Note: Make switch when P > Rand(0, 1)

38 Simulated Annealing - Sub Routines P: The probability function is most commonly solved for using The Metropolis-Hastings Algorithm { e (yc yn)/t y c y n P(y c, y n, t) = 1 y c > y n The graphs below show the relationship of P in terms of t with a fixed difference in y c and y n Note: Make switch when P > Rand(0, 1)

39 Simulated Annealing - Graphically

40 Questions?

41 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[

42 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[ Tabu

43 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[ Tabu, tapu

44 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[ Tabu, tapu, kapu

45 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[ Tabu, tapu, kapu and taboo are several anglican spelllings for the Polynesian word meaning sacred or forbidden

46 Search Algorithms - Tabu Search Tabu search, created by Fred W. Glover in 1986 and formalized in 1989, is a metaheuristic search method employing local search methods used for mathematical optimization.[ Tabu, tapu, kapu and taboo are several anglican spelllings for the Polynesian word meaning sacred or forbidden A tabu search differs from simulated annealing via the application of a tabu list. The tabu list is a list of not as good cites that have already been visited. The Neighbor function is designed to not return an item on the tabu list.

47 TSP - Example The Problem: Find the shortest path that visits all n cities exactly once. A Solution: Simulated Annealing

48 TSP - Example Process

49 TSP - Example Process 1 st : Outline problem and model scenario

50 TSP - Example Process 1 st : Outline problem and model scenario 2 nd : Write pseudo code for Temp function

51 TSP - Example Process 1 st : Outline problem and model scenario 2 nd : Write pseudo code for Temp function 3 rd : Write pseudo code for Neighbor function

52 TSP - Example Process 1 st : Outline problem and model scenario 2 nd : Write pseudo code for Temp function 3 rd : Write pseudo code for Neighbor function 4 th : Consider writing it in the programming language of your choice

53 TSP - Example Outline Problem

54 TSP - Example Outline Problem Align cities as points on the cartesian plane

55 TSP - Example Outline Problem Align cities as points on the cartesian plane

56 TSP - Example Outline Problem Align cities as points on the cartesian plane Define a variable which represents the route taken

57 TSP - Example Outline Problem Align cities as points on the cartesian plane Define a variable which represents the route taken Let R n be the set of our n ordered duples that represent our map. The r 1 element represents the starting point of the route and the r n be the last stop.

58 TSP - Example Outline Problem Align cities as points on the cartesian plane Define a variable which represents the route taken Let R n be the set of our n ordered duples that represent our map. The r 1 element represents the starting point of the route and the r n be the last stop. Define a function that represents the length of the route taken

59 TSP - Example Outline Problem Align cities as points on the cartesian plane Define a variable which represents the route taken Let R n be the set of our n ordered duples that represent our map. The r 1 element represents the starting point of the route and the r n be the last stop. Define a function that represents the length of the route taken n 1 d(r i, r i+1 ) d(r a, r b ) = i=1 (r ax r bx ) 2 + ( r ay r by ) 2

60 TSP - Example Temp Function For sake of simplicity and sanity we ll select a linear function with an initial temperature of 1000 and a cooling rate of 1000 stop degrees per iteration. Temp (lcv, stop) = stop lcv

61 TSP - Example Temp Function For sake of simplicity and sanity we ll select a linear function with an initial temperature of 1000 and a cooling rate of 1000 stop degrees per iteration. Temp (lcv, stop) = stop lcv Temp (lcv, stop) = t 0 e lcv estop

62 TSP - Example Neighbor Function

63 TSP - Example Neighbor Function Each route is like an ordered list, therefore to alter the order we must swap the position of items.

64 TSP - Example Neighbor Function Each route is like an ordered list, therefore to alter the order we must swap the position of items. Constant sized interval

65 TSP - Example Neighbor Function Each route is like an ordered list, therefore to alter the order we must swap the position of items. Constant sized interval Randomly select two integer values between 1 and n, inclusive, called a and b. Let r t r a, r a r b and r b r t

66 TSP - Example Consideration

67 Questions?

68 Thank You

69 References mathworld.wolfram.com Carr, Roger. Simulated Annealing. From MathWorld A Wolfram Web Resource, created by Eric W. Weisstein.