Lipschitzian Optimization, DIRECT Algorithm, and Applications

Transcription

1 ,, and April 1, 2008,, and

2 Outline 1 2 3,, and

3 Outline 1 2 3,, and

4 Function Optimization Problem For a function f : D R d R, find min f(x). x D Simple Bounds Mostly we will assume l i x i u i for all i [d], i.e. every variable x i has some lower bound l i and some upper bound u i. This means D is a hyperrectangle.,, and

5 Taxonomy of Methods,, and

6 Shubert (1972) A Sequential Method Seeking the Global Maximum of a Function Definition A function f : D R d R is called Lipschitz-continuous if there exists a positive constant K R + such that f(x) f(x ) K x x, x, x D. Problem We consider the following minimization problem min f(x), x D where f is Lipschitz-continuous, and D simple bounded.,, and

7 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

8 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

9 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

10 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

11 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

12 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

13 Shubert s Algorithm in 1D If we substitute a and b for x into the definition of Lipschitz-continuity we get the following two conditions for f(x), where x [a, b], f(x) f(x) f(a) K(x a), f(b) + K(x b).,, and

14 Global vs. Local Search X(a, b, f, K) = a + b f(a) f(b) +, 2 2K f(a) + f(b) K(b a) B(a, b, f, K) =. 2 2,, and

15 Global vs. Local Search X(a, b, f, K) = a + b f(a) f(b) +, 2 2K f(a) + f(b) K(b a) B(a, b, f, K) =. 2 2,, and

16 Global vs. Local Search X(a, b, f, K) = a + b f(a) f(b) +, 2 2K f(a) + f(b) K(b a) B(a, b, f, K) =. 2 2,, and

17 Pros and Cons of Pros + Global search possible + Deterministic, no need for multiple runs + Few paramters apart from K no need for fine-tuning + K gives bound on error, no need to rely on arbitrary stopping criteria such as the number of iterations Cons (of Shubert s algorithm) - Lipschitz constant has to be known - Speed of convergence (global vs. local) - Computational complexity in higher dimensions,, and

18 Outline 1 2 3,, and

19 Problems of Liptschitzian Optimization Problem 1: Specifying K K might not be easily accessible. DIRECT needs no prior knowledge and uses all possible constants. Sounds terrific, but how... Problem 2: Convergence Speed The parameter K is a trade-off between global and local search. By using all possible K, DIRECT balances better between global and local search. Problem 3: Combinatorial Complexity in Higher Dimensions is initialized by evaluating the function at the corners of a hyperrectangle. We have to make O(2 d ) evaluations.,, and

20 DIRECT in 1D Jones, Perttunen, Stuckman (1993) Without the Lipschitz Constant The name DIRECT stands fro DIviding RECTangles, but also captures the fact that it is a direct search technique. Key idea: Sample the function at center of rectangle.,, and

21 Division of Intervals When dividing the search space we have to make sure that previous function evaluations are not lost, i.e. they are still at the center of some interval. Instead of a bisection we do a trisection.,, and

22 Lipschitz Bound f(x) f(c) + K(x c) for x c, f(x) f(c) K(x c) for x c.,, and

23 Potentially Optimal Intervals Let S be the the partition of [a, b] into subintervals, S = m. Definition An interval j S is called potentially optimal if there exists some constant K 0 such that the following conditions hold, f(c j ) K((b j a j )/2) f(c i ) K((b i a i )/2) i S, (1) f(c j ) K((b j a j )/2) f min ǫ f min, (2) where ǫ 0.,, and

24 Potentially Optimal Intervals (1): f(c j ) K((b j a j )/2) f(c i ) K((b i a i )/2), i S,, and

25 Potentially Optimal Intervals (1): f(c j ) K((b j a j )/2) f(c i ) K((b i a i )/2), i S,, and

26 Potentially Optimal Intervals (1): f(c j ) K((b j a j )/2) f(c i ) K((b i a i )/2), i S,, and

27 Potentially Optimal Intervals (1): f(c j ) K((b j a j )/2) f(c i ) K((b i a i )/2), i S,, and

28 Potentially Optimal Intervals (2): f(c j ) K((b j a j )/2) f min ǫ f min,, and

29 Potentially Optimal Intervals (2): f(c j ) K((b j a j )/2) f min ǫ f min,, and

30 Summary DIRECT in 1D Input : a, b R, f( ), ǫ 0 Output: f min Initialize; repeat Identify set S of potentially optimal intervals; for s S do Evaluate new center points and subdivide s; until too many iterations ; return f min ;,, and

31 Division of Hypercubes 1. Evaluate f at c ± δe i, where e i is the i th unit vector. 2. Subdivide along directions with best function values first. This way the largest rectangles contain the best function values.,, and

32 Division of Hypercubes 1. Evaluate f at c ± δe i, where e i is the i th unit vector. 2. Subdivide along directions with best function values first. This way the largest rectangles contain the best function values.,, and

33 Division of Hypercubes 1. Evaluate f at c ± δe i, where e i is the i th unit vector. 2. Subdivide along directions with best function values first. This way the largest rectangles contain the best function values.,, and

34 Division of Hypercubes 1. Evaluate f at c ± δe i, where e i is the i th unit vector. 2. Subdivide along directions with best function values first. This way the largest rectangles contain the best function values.,, and

35 Division of Hyperrectangles Only divide along the set of longest sides. Rectangles have side lengths either 3 k or 3 (k+1), for k N. This fact is essential for the convergence of DIRECT.,, and

36 Division of Hyperrectangles Only divide along the set of longest sides. Rectangles have side lengths either 3 k or 3 (k+1), for k N. This fact is essential for the convergence of DIRECT.,, and

37 Convergence of DIRECT Theorem (Jones, Perttunen, Stuckman, 1993) DIRECT samples a dense subset of the unit cube, i.e. for any point x in the unit hypercube and δ > 0, DIRECT will eventually sample a point y such that x y 2 δ. Proof Let D be the d-dimensional unit hypercube. A rectangle R that has been involved in r divisions will have j := r mod d sides of length 3 (k+1) and d j sides of length 3 k, where k = (r j)/d. The radius of R is therefore (j3 2(k+1) + (d j)3 2k )/2, which goes to zero as r approches infinity. Let t N be the current iteration, and r t N the fewest number of divisions undergone by any rectangle.,, and

38 Convergence of DIRECT Proof (cont d) Claim: lim t r t =. Assume otherwise: t after which r t never changes, i.e. lim t r t = r t. After iteration t there will be a finite number of rectangles (say N) of maximal size. The one with the lowest function value will be potentially optimal, and therefore subdivided. This only leaves N 1 maximal rectangles. After N 1 iterations r t has increased by 1. Corollary (Jones, Perttunen, Stuckman, 1993) If the function f is continuous in the neigborhood of f := min x D f(x) then DIRECT converges to f.,, and

39 Convergence of DIRECT Finkel, Kelley (2004) Convergence Analysis of the, application of nonsmooth analysis. Definition The generalized directional derivative of f at x D in direction v is f 0 (x, v) := lim sup y x,y D, t 0,y+tv D f(y + tv) f(y). t Theorem (Finkel, Kelley, 2004) If f is Lipschitz-continuous on D and x is a cluster point of the sequence of DIRECT s best points, then f 0 (x, v) 0.,, and

40 Speed Up for Easy Functions Gablonsky, Kelley (2001) A Locally-Biased Form of The If there are only a few local minima, DIRECT uses a lot of unnecessary time exploring unvisited territory. Idea: Group rectangles by L norm, i.e. by their longest side and not their diameter. This leads to reduction in the number of groups, especially in the large and unimportant rectangles. Little theoretical, but at least some experimental evidence that this scheme works.,, and

41 Additive Scaling Problem Finkel, Kelley (2006) Additive Scaling and the Theorem (Finkel, Kelley, 2006) Let R be a hypercube sampled by DIRECT, and α(r) its size. Suppose that R is in the set of the smallest rectangles and that f(c(r)) = f min. If ǫ > 2α(R)K f(c(r)) ( (d + 8) d), then R will not be subdivided until all rectangles in its neighborhood are of the same size as R.,, and

42 Additive Scaling Problem Theorem (Finkel, Kelley, 2006) Let R be a hypercube sampled by DIRECT. Suppose there exists a hyperrectangle T such that α(t) > α(r). If f min > then R will not be subdivided. K d ǫ( 1 + 8/d 1), The authors propose a variant of the definition of potential optimality, f(c R ) Kα(R) f min ǫ f min f median. Experimental results that the modified DIRECT is stable under additive scaling.,, and

43 Outline 1 2 3,, and

44 Aircraft Routing Bartholomew-Biggs, Parkhurst, Wilson (2002) Using DIRECT to Solve an Aircraft Routing Problem Two-dimensional (euclidian) shortest path problem subject to obstacle regions, rendezvous time, speed and maneuverability constraints, visibility, etc... Optimization of a multivariate non-differentiable function. Therefore, DIRECT seems to be a good approach. Waypoints are variables of the function to optimize. Restart slightly improves performance.,, and

45 Component Design Zhu, Bogy (2002) and Its Application to Slider Air-Bearing Surface Optimization, design of hard-drive heads. Shape of the magnetic head is crucial to movement. Height above the disk is referred to as flying height. DIRECT outperforms Simulated Annealing. Better convergence rate and better result.,, and

46 Last Slide Open semester thesis: Global Function Optimization for Aircraft Routing, Implementation of DIRECT in C++. Thank you for your attention.,, and