Artificial Intelligence, CS, Nanjing University Spring, 2016, Yang Yu. Lecture 5: Search 4.

Transcription

1 Artificial Intelligence, CS, Nanjing University Spring, 2016, Yang Yu Lecture 5: Search 4

2 Previously... Path-based search Uninformed search Depth-first, breadth first, uniform-cost search Informed search Best-first, A* search Adversarial search Alpha-Beta search

3 Beyond classical search Bandit search Tree search: Monte-Carlo Tree Search General search: Gradient decent Metaheuristic search

4 Bandit Multiple arms Each arm has an expected reward, but unknown, with an unknown distribution Maximize your award in fixed trials

5 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem?

6 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem? waste on suboptimal arms

7 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem? waste on suboptimal arms Exploitation-only:

8 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem? waste on suboptimal arms Exploitation-only: 1. try each arm once 2. try the observed best arm T-K times

9 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem? waste on suboptimal arms Exploitation-only: 1. try each arm once 2. try the observed best arm T-K times problem?

10 Simplest strategies Two simplest strategies Exploration-only: for T trails and K arms, try each arm T/K times problem? waste on suboptimal arms Exploitation-only: 1. try each arm once 2. try the observed best arm T-K times problem? risk of wrong best arm

11 ε-greedy Balance the exploration and exploitation: with ε probability, try a random arm with 1-ε probability, try the best arm ε controls the balance

12 Softmax Balance the exploration and exploitation: Choose arm with probability τ controls the balance

13 Upper-confidence bound Balance the exploration and exploitation: Choose arm with the largest value of average reward + upper confidence bound 2.5% Q(k) UCB

14 Monte-Carlo Tree Search Gradually grow the search tree: I Iterate Tree-Walk I Building Blocks I I I I I I Returned solution: I Select next action Bandit phase Add a node Grow a leaf of the search tree Select next action bis Random phase, roll-out Compute instant reward Evaluate Update information in visited nodes Propagate Path visited most often Kocsis Szepesvári, 06 Bandit Based Phase Search Tree Random Phase Explored Tree New Node

15 Monte-Carlo Tree Search Example: Pic from How to select the leave? As bandit rollout

16 Monte-Carlo Tree Search codes from

17 Monte-Carlo Tree Search codes from

18 Monte-Carlo Tree Search optimal? Yes, after infinite tries compare with alpha-beta pruning no need of heuristic function

19 Monte-Carlo Tree Search Improving random rollout Monte-Carlo-based Brügman Until the goban is filled, add a stone (black or white in turn) at a uniformly selected empty position 2. Compute r =Win(black) 3. The outcome of the tree-walk is r Improvements? I Put stones randomly in the neighborhood of a previous stone I Put stones matching patterns prior knowledge I Put stones optimizing a value function Silver et al. 07

20 General search

21 Greedy idea in continuous space Suppose we want to site three airports in Romania: 6-D state space defined by (x 1, y 2 ), (x 2, y 2 ), (x 3, y 3 ) objective function f(x 1, y 2, x 2, y 2, x 3, y 3 ) = sum of squared distances from each city to nearest airport 71 Oradea Neamt Zerind 75 Arad Timisoara Dobreta 151 Sibiu 99 Fagaras 80 Rimnicu Vilcea Lugoj Mehadia 120 Pitesti Bucharest Craiova 90 Giurgiu 87 Iasi Urziceni Vaslui Hirsova 86 Eforie

22 Greedy idea in continuous space discretize and use hill climbing 71 Oradea Neamt Zerind 75 Arad Timisoara Dobreta 151 Sibiu 99 Fagaras 80 Rimnicu Vilcea Lugoj Mehadia 120 Pitesti Bucharest Craiova 90 Giurgiu 87 Iasi Urziceni Vaslui Hirsova 86 Eforie

23 Greedy idea in continuous space gradient decent 6-D state space defined by (x 1, y 2 ), (x 2, y 2 ), (x 3, y 3 ) objective function f(x 1, y 2, x 2, y 2, x 3, y 3 ) = sum of squared distances from each city to nearest airport Gradient methods compute f = ± f, f, f, f, f, f x 1 y 1 x 2 y 2 x 3 y 3 to increase/reduce f, e.g., by x x + α f(x) 1-order method

24 Greedy idea in continuous space gradient decent 6-D state space defined by (x 1, y 2 ), (x 2, y 2 ), (x 3, y 3 ) objective function f(x 1, y 2, x 2, y 2, x 3, y 3 ) = sum of squared distances from each city to nearest airport Sometimes can solve for f(x) = 0 exactly (e.g., with one city). Newton Raphson (1664, 1690) iterates x x H 1 f (x) f(x) to solve f(x) = 0, where H ij = 2 f/ x i x j 2-order method Taylor s series: f(x) =f(a)+(x a)f (a)+ (x a)2 2 f (a)+ = i=0 (x a) i f (i) (a). i!

25 Greedy idea 1st and 2nd order methods may not find global optimal solutions they work for convex functions objective function global maximum shoulder local maximum "flat" local maximum current state state space

26 Meta-heuristics problem independent black-box zeroth-order method... and usually inspired from nature phenomenon

27 Simulated annealing temperature from high to low when high temperature, form the shape when low temperature, polish the detail

28 Simulated annealing Idea: escape local maxima by allowing some bad moves but gradually decrease their size and frequency function Simulated-Annealing( problem, schedule) returns a solution state inputs: problem, a problem schedule, a mapping from time to temperature local variables: current, a node next, a node T, a temperature controlling prob. of downward steps current Make-Node(Initial-State[problem]) for t 1 to do T schedule[t] if T = 0 then return current next a randomly selected successor of current E Value[next] Value[current] if E > 0 then current next else current next only with probability e E/T the neighborhood range shrinks with T the probability of accepting a bad solution decreases with T

29 Simulated annealing a demo graphic from

30 Local beam search Idea: keep k states instead of 1; choose top k of all their successors Not the same as k searches run in parallel! Searches that find good states recruit other searches to join them Problem: quite often, all k states end up on same local hill Idea: choose k successors randomly, biased towards good ones Observe the close analogy to natural selection!

31 Genetic algorithm a simulation of Darwin s evolutionary theory over-reproduction with diversity nature selection random initialization parent population reproduction offspring solutions selection evaluated offspring solutions evaluation

32 Genetic algorithm Encode a solution as a vector, 1: Pop n randomly drawn solutions from X 2: for t=1,2,... do 3: Pop m {mutate(s) 8s 2 Pop}, the mutated solutions 4: Pop c {crossover(s 1,s 2 ) 9s 1,s 2 2 Pop m }, the recombined solutions 5: evaluate every solution in Pop c by f(s)(8s 2 Pop c ) 6: Pop s selected solutions from Pop and Pop c 7: Pop Pop s 8: terminate if meets a stopping criterion 9: end for

33 Genetic algorithm % % % % Fitness Selection Pairs Cross Over Mutation GAs require states encoded as strings (GPs use programs) Crossover helps iff substrings are meaningful components + =

34 Example Encode a solution as a vector with length n each element of the vector can be chosen from {1,...,V } parameters: mutation probability pm, crossover probability pc 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: Pop = randomly generate n solutions from {1,...,V } n for t=1,2,... do Pop m =emptyset, Pop c =emptyset for i = 1 to n let x be the i-th solution in Pop for j = 1 to n: with probability p m, change x j by a random value from {1,...,V } add x into Pop m end for for i = 1 to n let x be the i-th solution in Pop m let x be a randomly selected solution from Pop m with probability p c, exchange a random part of x with x add x into Pop c end for evaluate solutions in Pop c, select the best n solutions from Pop and Pop c to Pop terminal if a good solution is found end for

35 An evolutionary of virtual life

36 An evolutionary of virtual life

37 Properties of meta-heuristics zeroth order do not need differentiable functions convergence will find an optimal solution if or P( x -> x1 ->... -> xk -> x* )>0 P( x* x )>0 a missing link observation simulation

38 Properties of meta-heuristics zeroth order do not need differentiable functions convergence will find an optimal solution if or P( x -> x1 ->... -> xk -> x* )>0 P( x* x )>0 a missing link observation simulation observation principle application

39 Properties of meta-heuristics grey wolf optimizer 2010 gravitational search algorithm river formation dynamics 2000 differential evolution 1990 memetic algorithms cultural algorithms 1980 fireworks algorithm brainstorm algorithm bat algorithm intelligent water drops algorithm artificial bee colony algorithms particle swarm optimization algorithms ant colony optimization algorithms artificial immune systems tabu search simulated annealing 1970 evolutionary strategies evolutionary programming 1960 genetic algorithms

40 Example hard to apply traditional optimization methods but easy to test a given solution Representation: parameterize Fitness: represented as a vector of parameters x i f(x i ) test by simulation/experiment

41 Example this nose... has been newly developed... using the latest analytical technique (i.e. genetic algorithms) N700 cars save 19% energy... 30% increase in the output... This is a result of adopting the... nose shape

42 Example NASA ST5 hard to apply traditional optimization methods but easy to test a given solution

43 Example NASA ST5 hard to apply traditional optimization methods but easy to test a given solution

44 Example NASA ST5 hard to apply traditional optimization methods but easy to test a given solution تج) QHAs % $ ) 38% efficiency evolved antennas resulted in 93% efficiency

45 Different Environment Properties

46 Nondeterministic actions ose that we introduce nondeterminism in the form of a powerful In the erratic vacuum world, thesuck action works as follows: When applied to a dirty square the action cleans the square and sometimes cleans up dirt in an adjacent square, too. When applied to a clean square the action sometimes deposits dirt on the carpet almost all real-world problems are nondeterministic how do you solve this problem?

47 AND-OR tree search OR node: different actions (as usual) AND node: different transitions 1 Suck Right GOAL Suck Right Left Suck 5 LOOP 1 LOOP 6 Suck Left 1 LOOP 8 4 GOAL 8 GOAL 5 LOOP a solution is not a path but a tree

48 Depth-first AND-OR tree search function AND-OR-GRAPH-SEARCH(problem) returns a conditional plan, or failure OR-SEARCH(problem.INITIAL-STATE, problem, []) function OR-SEARCH(state, problem, path) returns a conditional plan, or failure if problem.goal-test(state) then return the empty plan if state is on path then return failure for each action in problem.actions(state) do plan AND-SEARCH(RESULTS(state, action), problem, [state path]) if plan failure then return [action plan] return failure function AND-SEARCH(states, problem, path) returns a conditional plan, or failure for each s i in states do plan i OR-SEARCH(s i, problem, path) if plan i = failure then return failure return [if s 1 then plan 1 else if s 2 then plan 2 else...if s n 1 then plan n 1 else plan n ]

49 Search with no observations search in belief (in agent s mind) L R L R S S S L R 5 S 5 3 L 6 4 S 4 R 7 L 7 R 8 8 L R 6 8 S 8 7 L R S 3 7