2. Optimization problems 6

Transcription

1 6 2.1 Examples Convex sets and functions Convex optimization problems Examples 7-1 An (NP-) optimization problem P 0 is defined as follows Each instance I P 0 has a feasibility domain S I. Its elements y S I are called solutions Feasibility (y S I ) can in be tested in polynomial time Task: Given an instance I and an objective function c : S I -> Q (rational numbers), find an optimal solution y = OPT(I) i.e., y S I with OPT(I) = c(y)! c(x) for all x S I OPT(I) denotes both, the optimal solution and the objective function value of the optimal solution such a solution is called a global optimum (global minimum) or optimum (minimum) An algorithm that does this is called exact 2.1 Example: Traveling Salesman Problem (TSP) Instance Complete Graph K n, n " 3 Rational edge weights c(e) " 0 Task

2 2.1 Examples 7-2 Compute a Hamiltonian cycle C with minimal length!!"" # # $!""!!#" A concrete instance: G = K n in the plane with Euclidean distances Two feasible solutions 8-1 Neighborhoods Neighborhoods are defined as!-niggarded (w.r.t. some norm) for continuous problems. How for discrete problems? A neighborhood for a problem class P 0 is given by a mapping! "! # " " # " for each instance I P 0 N I (y) is called the neighborhood of y S I. We write N(y) of I is clear from the context 2.4 Example: TSP Define a neighborhood by a 2-exchange N 2 (y) := { x S I x results from y by exchanging! 2 edges from y by other edges }

3 8-2 This generalizes to any k " 2 and yields the neighborhood N k (y) 2.5 Example: MST Define a neighborhood by exchanging an edge on a fundamental cycle N(y) := { x S I x results from y by adding an edge to y and deleting another edge on the resulting cycle} 8-3 G T T' 2.6 Example: LP Define a neighborhood as!-neighborhood N! (y) := { x Ax = b, x " 0, y-x!! } y! Local and global optima

4 8-4 Local and global optima Consider a problem class P 0 with neighborhood N and let I P 0 y S I is called locally optimal w.r.t. N if c(y)! c(x) for all x N I (y) 2.7 Example: local minima in calculus 2.8 Example: TSP Locally optimal solutions w.r.t. N k are called k-optimal or k-opt for short exact neighborhood A neighborhood N for a problem class P 0 is called exact :<=> every local optimum w.r.t. N is a global optimum 8-5 :<=> every local optimum w.r.t. N is a global optimum More precisely, for all I P 0, every locally optimal y S I w.r.t. N I is globally optimal 2.9 Example: TSP N 2 is not exact Counter example : 1 cost a 5 2 cost b cost c 4 3 a < b < c tour y = "outer edges" has cost 5b Since both green edges are adjacent, every 2-exchange can at most add one green edge, thus at best one green and one red The new tour is worse if a+c > 2b => y is locally optimal w.r.t. N

5 8-6 => y is locally optimal w.r.t. N 2 Two successive 2-exchanges add both green edges to the tour This tour is better than y if 2a+c < 3b a < b < c and a+c > 2b and 2a+c < 3b are fulfilled by a = 1, b = 4, c = 8 N n is exact clear since N n (y) = S I! 2.10 Example: MST The neighborhood of MST is exact Proof : 8-7 Proof : Use a theorem from ADM I: T is optimal <=> every non-tree-edge e is the most expensive edge in the cycle induced by e in T + e! Neighborhoods motivate the principle of local search Algorithm local search Input instance I of an optimization problem P 0 with neighborhood N I start solution y S I Output local optimum w.r.t. N I Method iterative improvement while there is a better solution x N I (y) do choose better solution x N I (y) y := x

6 8-8 y := x return y 2.11 Theorem (local search for MST) Local search w.r.t. the MST-neighborhood is a polynomial algorithm for computing a (globally) optimal MST if (a) it always chooses a non-tree-edge f that is cheaper than the most expensive edge of the cycle K induced by f (b) it always deletes a most expensive tree-edge e from the cycle K induced by f Proof: 1. Since the neighborhood is exact, the algorithm has computed a globally optimal solution at termination 2. The algorithm terminates in polynomial time Claim 1: A deleted edge never returns into the tree Proof by contradiction Let K be the cycle when edge e is removed 8-9 e K Consider the first later point in time t at which e is chosen to enter the tree => e is currently a non-tree-edge and induces a cycle K' e K K' => K' results from K by the local search steps until time t In every of these steps, e is a non-tree-edge and induces a cycle K(e) in the current tree Claim 2: In every of these steps, c(e) " c(g) for all edges g K(e) Proof by induction along the sequence of cycles K = K 1, K 2, K 3,... Base case: clear for K = K 1 by construction Inductive step from K i to K i+1 clear for K = K

7 8-10 clear for K i = K i+1 let K i # K i+1 two cases are possible K i+1 makes K i larger => K i+1 = K i - (some edges including the currently deleted edge e') + P (P is part of the current cycle from which we delete e') P e e e' => c(e') " c(f) for all edges f P because of (b) c(e) " c(e') by the inductive assumption, (b) => c(e) " c(f) for all edges f K i+1 K i+1 makes K i smaller => K i+1 = K + (some edges including the currently added edge e') - P (P is part of K i ) 8-11 f 0 P e e e' => c(e') < c(f 0 ) for the removed edge f 0, c(e) " c(f) for all edges f P because of the inductive assumption => c(e) " c(f) for all edges f K i+1 Claim 2 contradicts the choice of e specified in (a) So there are at most m-n+1 exchanges (m = # edges, n = # vertices). Every exchange step can be done in O(n) time determine the most expensive edge e in the cycle induced by the non-tree-edge f and compare c(e) and c(f) [by breadth first search in O(# edges in tree) = O(n)] exchange e and f if c(e) < c(f) [O(n) if the tree is maintained as array of adjacency lists]

8 8-12 [O(n) if the tree is maintained as array of adjacency lists] Thus O((m-n+1)n) = O(mn) altogether! Remark: we have obtained better algorithms in ADM I: Kruskal O(m log n) and Prim O(n 2 ) Exercises: Analyze local search for TSP with the k-opt neighborhood. Does it run in polynomial time? 2.3 Convex sets and functions 9-1 Convex combination of two vectors Let x, y R n. Then every point z = λ x + (1-λ) y with 0! λ! 1 is called a convex combination of x and y (a strictly convex combination if 0 < λ < 1) the convex combinations of x and y are exactly the points on the line segment from x to y x-y x y points on this line segment are vectors of the form y + λ(x-y) Convex set S! R n is called convex if S contains all convex combinations of any two points x, y S 2.12 Example:

9 2.3 Convex sets and functions 9-2 R n, Ø, {x}, x R n are convex The convex subsets of R 1 are precisely the intervals Convex sets in R 2 are those without "bays" A B C 2.13 Lemma The intersection of (any number of) convex sets is convex Proof: Let S = " i I S i, S i convex Let x, y S, 0! λ! 1, z = λ x + (1-λ) y Def. of S => x,y S i for all i => z S i for all i => z S! Lemma 2.13 is the basis for the definition of the convex hull of a set 2.3 Convex sets and functions 9-3 The convex hull conv(s) of a set S is the smallest convex set containing S, i.e.,!"#$%!& ' "! "# " ("#$)* This intersection exists, since R n is one of the sets M An equivalent description is (exercise) conv(s) = { λ 1 x λ k x k x i S, λ i " 0, $ λ i = 1, k finite } Theorem of Caratheodory: in R n, k! n+1 suffices Convex function Let S! R n be a convex set. A function c : S -> R 1 is called convex in S if c(λ x + (1-λ) y)! λ c(x) + (1-λ) c(y) for all x, y S, all 0! λ! Example Every linear function is convex Interpretation of convex functions c : R 1 -> R 1

10 2.3 Convex sets and functions 9-4 c(y)!"c(x) + (1-!) c(y) c(x) c x z y z := λ x + (1-λ) y c(z)! λ c(x) + (1-λ) c(y) 2.15 Lemma Let c be convex in S! R n. Then, for every real number t, the level set S t := { x S c(x)! t } is convex Proof: Consider z := λ x + (1-λ) y with x, y S t, 0! λ! 1 => c(z)! λ c(x) + (1-λ) c(y) since c is convex => z S t!! λ t + (1-λ) t since x, y S t Level sets of a convex function c : R 2 -> R Convex sets and functions 9-5 Level sets of a convex function c : R 2 -> R 1 c = Concave function c in S! R n is called concave if -c is convex <=> c(λ x + (1-λ) y) " λ c(x) + (1-λ) c(y) for all x, y S, all 0! λ! 1

11 2.4 Convex optimization problems 10-1 Convex optimization = minimizing a convex function on a convex set. Important principle: local Optima are global Optima 2.16 Theorem (local - global) Consider an instance I of an optimization problem with S I! R n convex and c convex in S I. => The neighborhood N! (y) := { x S I : y-x!! } defined by the Euclidean distance is exact for every! > 0. Proof: Let! > 0 and let y be locally optimal w.r.t. N!. Consider x S I. Show that c(y)! c(x). This is trivial if x N! (y) So assume x N! (y) => then there is some λ such that z := λ x + (1-λ) y N! (y) and z # y 2.4 Convex optimization problems 10-2 S I x z N! (y) y c convex => c(z)! λ c(x) + (1-λ) c(y), moreover c(z) " c(y) since y is locally optimal!!#" "!# " $"!!%"!!%" "!# " $"!!%"!!""!! $!!%" $ $ Hence c(x) " c(y)! Observe: this holds without any further assumptions on c; in particular, c need not be differentiable. Historical definition of convex optimization problems An instance I of an optimization problem is called a convex optimization problem if S is specified as set of all x R n fulfilling side constraints of the form:

12 2.4 Convex optimization problems 10-3 S I is specified as set of all x R n fulfilling side constraints of the form: g i (x) " 0 g i : R n -> R 1 concave, i = 1,...,m i = 1,...,m c is convex in S I 2.17 Lemma The feasible set S I of an instance I of a historically defined convex optimization problem is convex Proof: g i concave => -g i convex => S i := { x -g i (x)! 0 } = { x g i (x) " 0 } convex because of Lemma 2.15 => S I = " i S i is convex because of Lemma 2.13! 2.18 Theorem In a convex optimization problem, every local optimum is a global optimum 2.19 Remark There can be many global optima 2.4 Convex optimization problems 10-4 There can be many global optima Every instance of LP is a convex optimization problem => every local minimum is a global minimum Calculus offers sufficient criteria for smooth functions to be convex: D! R n open, c : D -> R 1 is twice continuously differentiable, Hessian matrix (= matrix of 2nd partial derivatives) of c is positive semidefinite