The Cross-Entropy Method for Mathematical Programming

Size: px

Start display at page:

Download "The Cross-Entropy Method for Mathematical Programming"

Kristian Holland
5 years ago
Views:

1 The Cross-Entropy Method for Mathematical Programming Dirk P. Kroese Reuven Y. Rubinstein Department of Mathematics, The University of Queensland, Australia Faculty of Industrial Engineering and Management, Technion, Israel The Cross-Entropy Method formathematical Programming p. 1/42

2 Contents 1. Introduction 2. CE Methodology 3. Applications 4. Some Theory on CE 5. Conclusion + Discussion The Cross-Entropy Method formathematical Programming p. 2/42

3 CE Matters Book: R.Y. Rubinstein and D.P. Kroese. The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte Carlo Simulation and Machine Learning, Springer-Verlag, New York, The Cross-Entropy Method formathematical Programming p. 3/42

4 CE Matters Book: R.Y. Rubinstein and D.P. Kroese. The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte Carlo Simulation and Machine Learning, Springer-Verlag, New York, Special Issue: Annals of Operations Research (Jan 2005). The Cross-Entropy Method formathematical Programming p. 3/42

5 CE Matters Book: R.Y. Rubinstein and D.P. Kroese. The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte Carlo Simulation and Machine Learning, Springer-Verlag, New York, Special Issue: Annals of Operations Research (Jan 2005). The CE home page: The Cross-Entropy Method formathematical Programming p. 3/42

6 Introduction The Cross-Entropy Method was originally developed as a simulation method for the estimation of rare event probabilities: Estimate P(S(X) γ) X: random vector/process taking values in some set X. S: real-valued function on X. The Cross-Entropy Method formathematical Programming p. 4/42

7 Introduction The Cross-Entropy Method was originally developed as a simulation method for the estimation of rare event probabilities: Estimate P(S(X) γ) X: random vector/process taking values in some set X. S: real-valued function on X. It was soon realised that the CE Method could also be used as an optimization method: Determine max x X S(x) The Cross-Entropy Method formathematical Programming p. 4/42

8 Some Applications Combinatorial Optimization (e.g., Travelling Salesman, Maximal Cut and Quadratic Assignment Problems) Noisy Optimization (e.g., Buffer Allocation, Financial Engineering) Multi-Extremal Continuous Optimization Pattern Recognition, Clustering and Image Analysis Production Lines and Project Management Network Reliability Estimation Vehicle Routing and Scheduling DNA Sequence Alignment The Cross-Entropy Method formathematical Programming p. 5/42

9 A Multi-extremal Function The Cross-Entropy Method formathematical Programming p. 6/42

10 A Maze Problem The Optimal Trajectory The Cross-Entropy Method formathematical Programming p. 7/42

11 A Maze Problem Iteration 1: The Cross-Entropy Method formathematical Programming p. 8/42

12 A Maze Problem Iteration 2: The Cross-Entropy Method formathematical Programming p. 9/42

13 A Maze Problem Iteration 3: The Cross-Entropy Method formathematical Programming p. 10/42

14 A Maze Problem Iteration 4: The Cross-Entropy Method formathematical Programming p. 11/42

15 The Optimisation Setting Many real-world optimisation problems can be formulated mathematically as either (or both) 1. Locate some element x in a set X such that S(x ) S(x) for all x X, where S is a objective function or performance measure defined on X. 2. Find γ = S(x ), the globally maximal value of the function. The Cross-Entropy Method formathematical Programming p. 12/42

16 The Optimisation Setting Many real-world optimisation problems can be formulated mathematically as either (or both) 1. Locate some element x in a set X such that S(x ) S(x) for all x X, where S is a objective function or performance measure defined on X. 2. Find γ = S(x ), the globally maximal value of the function. X discrete: combinatorial optimisation problem X continuous: continuous optimisation problem. The Cross-Entropy Method formathematical Programming p. 12/42

17 A New Approach: CE Instead of locating optimal solutions to a particular problem directly, the CE method aims to locate an optimal sampling distribution The sampling distribution is optimal if only optimal solutions can be sampled from it. The Cross-Entropy Method formathematical Programming p. 13/42

18 General CE Procedure First, provide the state space X (whether the states are coordinates, permutations, or some other mathematical entities) over which the problem is defined, along with a performance measure S on this space. The Cross-Entropy Method formathematical Programming p. 13/42

19 General CE Procedure First, provide the state space X (whether the states are coordinates, permutations, or some other mathematical entities) over which the problem is defined, along with a performance measure S on this space. Second, formulate the parameterized sampling distribution to generate objects X X. Then, iterate the following steps: The Cross-Entropy Method formathematical Programming p. 13/42

20 General CE Procedure First, provide the state space X (whether the states are coordinates, permutations, or some other mathematical entities) over which the problem is defined, along with a performance measure S on this space. Second, formulate the parameterized sampling distribution to generate objects X X. Then, iterate the following steps: Generate a random sample of states X 1,..., X N X. The Cross-Entropy Method formathematical Programming p. 13/42

21 General CE Procedure First, provide the state space X (whether the states are coordinates, permutations, or some other mathematical entities) over which the problem is defined, along with a performance measure S on this space. Second, formulate the parameterized sampling distribution to generate objects X X. Then, iterate the following steps: Generate a random sample of states X 1,..., X N X. Update the parameters of the random mechanism (obtained via CE minimization), in order to produce a better sample in the next iteration. The Cross-Entropy Method formathematical Programming p. 13/42

22 General CE Algorithm 1. Initialise the parameters of the algorithm. 2. Generate a random sample from the sampling distribution. 3. Update the parameters of the sampling distribution on the basis of the best scoring samples, the so-called elite samples. This step involves the Cross-Entropy distance. 4. Repeat from Step 2 until some stopping criterion is met. The Cross-Entropy Method formathematical Programming p. 14/42

23 Degenerate Sampling Distribution Suppose that there is a unique solution x to the problem. Then, the degenerate distribution assigns all of the probability mass (in the discrete case) or density (in the continuous case) to this point in X. It is exactly the degenerate distribution from which we wish to ultimately sample, in order to obtain an optimal solution. The Cross-Entropy Method formathematical Programming p. 15/42

24 Detailed Generic CE Algorithm 1. Choose an initial parameter vector, v 0. Set t = Generate X 1, X 2,..., X N from the density f( ; v t 1 ) 3. Compute the sample (1 ρ)-quantile, ˆγ t = S ( (1 ρ)n ), locate the elite samples E = {X i : S(X i ) ˆγ t }, and determine ṽ t = argmax ln f(x i ; v). v X i E Set the current estimate for the optimal parameter vector, v t to: v t = αṽ t + (1 α)v t If the stopping criterion is met, stop; otherwise set t = t + 1, and return to Step 2. The Cross-Entropy Method formathematical Programming p. 16/42

25 Maximum Likelihood There is a connection between the Cross-Entropy method and Maximum Likelihood Estimation, which implies a very simple way of updating the parameter vector: take the maximum likelihood estimate of the parameter vector based on the elite samples. Thus, the sampling distribution comes closer at each iteration to the degenerate distribution, which is the optimal sampling distribution for the problem. The Cross-Entropy Method formathematical Programming p. 17/42

26 Example: The Max-Cut Problem Consider a weighted graph G with node set V = {1,..., n}. Partition the nodes of the graph into two subsets V 1 and V 2 such that the sum of the weights of the edges going from one subset to the other is maximised. Example: The Cross-Entropy Method formathematical Programming p. 18/42

27 Cost matrix (assume symmetric): 0 c 12 c c 21 0 c 23 c c 31 c 32 0 c 34 c 35 0 C = 0 c 42 c 43 0 c 45 c c 53 c 54 0 c c 64 c {V 1, V 2 } = {{1, 3, 4}, {2, 5, 6}} is a possible cut. The cost of the cut is c 12 + c 32 + c 35 + c 42 + c 45 + c 46. The Cross-Entropy Method formathematical Programming p. 19/42

28 Random Cut Vector We can represent a cut via a cut vector x = (x 1,..., x n ), where x i = 1 if node i belongs to same partition as 1, and 0 else. The Cross-Entropy Method formathematical Programming p. 20/42

29 Random Cut Vector We can represent a cut via a cut vector x = (x 1,..., x n ), where x i = 1 if node i belongs to same partition as 1, and 0 else. For example, the cut {{1, 3, 4}, {2, 5, 6}} can be represented via the cut vector (1, 0, 1, 1, 0, 0). The Cross-Entropy Method formathematical Programming p. 20/42

30 Random Cut Vector We can represent a cut via a cut vector x = (x 1,..., x n ), where x i = 1 if node i belongs to same partition as 1, and 0 else. For example, the cut {{1, 3, 4}, {2, 5, 6}} can be represented via the cut vector (1, 0, 1, 1, 0, 0). Let X be the set of all cut vectors x = (1, x 2,..., x n ). The Cross-Entropy Method formathematical Programming p. 20/42

31 Random Cut Vector We can represent a cut via a cut vector x = (x 1,..., x n ), where x i = 1 if node i belongs to same partition as 1, and 0 else. For example, the cut {{1, 3, 4}, {2, 5, 6}} can be represented via the cut vector (1, 0, 1, 1, 0, 0). Let X be the set of all cut vectors x = (1, x 2,..., x n ). Let S(x) be the corresponding cost of the cut. The Cross-Entropy Method formathematical Programming p. 20/42

32 Random Cut Vector We can represent a cut via a cut vector x = (x 1,..., x n ), where x i = 1 if node i belongs to same partition as 1, and 0 else. For example, the cut {{1, 3, 4}, {2, 5, 6}} can be represented via the cut vector (1, 0, 1, 1, 0, 0). Let X be the set of all cut vectors x = (1, x 2,..., x n ). Let S(x) be the corresponding cost of the cut. We wish to maximise S(x) via the CE method. The Cross-Entropy Method formathematical Programming p. 20/42

33 Generation and Updating Formulas Generation of cut vectors: The most natural and easiest way to generate the cut vectors X i = (1, X i1,..., X in ) is to let X i2,..., X in be independent Bernoulli random variables with success probabilities p 2,..., p n. The Cross-Entropy Method formathematical Programming p. 21/42

34 Generation and Updating Formulas Generation of cut vectors: The most natural and easiest way to generate the cut vectors X i = (1, X i1,..., X in ) is to let X i2,..., X in be independent Bernoulli random variables with success probabilities p 2,..., p n. Updating formulas: From CE minimization: the updated probabilities are the maximum likelihood estimates of the E = ρn elite samples: ˆp t,j = X i E X ij E, j = 2,..., n. The Cross-Entropy Method formathematical Programming p. 21/42

35 Algorithm 1 Start with ˆp 0 = (1, 1/2,..., 1/2). Let t := 1. The Cross-Entropy Method formathematical Programming p. 22/42

36 Algorithm 1 Start with ˆp 0 = (1, 1/2,..., 1/2). Let t := 1. 2 Generate: Draw X 1,..., X N from Ber(ˆp t ). Let ˆγ t be the worst performance of the elite performances. The Cross-Entropy Method formathematical Programming p. 22/42

37 Algorithm 1 Start with ˆp 0 = (1, 1/2,..., 1/2). Let t := 1. 2 Generate: Draw X 1,..., X N from Ber(ˆp t ). Let ˆγ t be the worst performance of the elite performances. 3 Update ˆp t : Use the same elite sample to calculate X ˆp t,j = i E X ij, j = 2,..., n, E where X i = (1, X i2,..., X in ), and increase t by 1. The Cross-Entropy Method formathematical Programming p. 22/42

38 Algorithm 1 Start with ˆp 0 = (1, 1/2,..., 1/2). Let t := 1. 2 Generate: Draw X 1,..., X N from Ber(ˆp t ). Let ˆγ t be the worst performance of the elite performances. 3 Update ˆp t : Use the same elite sample to calculate X ˆp t,j = i E X ij, j = 2,..., n, E where X i = (1, X i2,..., X in ), and increase t by 1. 4 If the stopping criterion is met, then stop; otherwise set t := t + 1 and reiterate from step 2. The Cross-Entropy Method formathematical Programming p. 22/42

39 Example Results for the case with n = 400, m = 200 nodes are given next. The Cross-Entropy Method formathematical Programming p. 23/42

40 Example Results for the case with n = 400, m = 200 nodes are given next. Parameters: ρ = 0.1, N = 1000 (thus E = 100). The Cross-Entropy Method formathematical Programming p. 23/42

41 Example Results for the case with n = 400, m = 200 nodes are given next. Parameters: ρ = 0.1, N = 1000 (thus E = 100). The CPU time was only 100 seconds (Matlab, pentium III, 500 Mhz). The Cross-Entropy Method formathematical Programming p. 23/42

42 Example Results for the case with n = 400, m = 200 nodes are given next. Parameters: ρ = 0.1, N = 1000 (thus E = 100). The CPU time was only 100 seconds (Matlab, pentium III, 500 Mhz). The CE algorithm converges quickly, yielding the exact optimal solution in 22 iterations. The Cross-Entropy Method formathematical Programming p. 23/42

43 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

44 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

45 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

46 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

47 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

48 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

49 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

50 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

51 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

52 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

53 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

54 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

55 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

56 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

57 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

58 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

59 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

60 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

61 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

62 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

63 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

64 Max-Cut The Cross-Entropy Method formathematical Programming p. 24/42

65 Example: Continuous Optimization 1 S(x) x 2 6 S(x) = e (x 2) e (x+2)2 The Cross-Entropy Method formathematical Programming p. 25/42

66 Matlab Program S = inline( exp(-(x-2).^2) + 0.8*exp(-(x+2).^2) ); mu = -10; sigma = 10; rho = 0.1; N = 100; eps = 1E-3; t=0; % iteration counter while sigma > eps t = t+1; x = mu + sigma*randn(n,1); SX = S(x); % Compute the performance. sortsx = sortrows([x SX],2); mu = mean(sortsx((1-rho)*n:n,1)); sigma = std(sortsx((1-rho)*n:n,1)); fprintf( %g %6.9f %6.9f %6.9f \n, t, S(mu),mu, sigma) end The Cross-Entropy Method formathematical Programming p. 26/42

67 Numerical Result Iteration 1 Iteration Iteration Iteration The Cross-Entropy Method formathematical Programming p. 27/42

68 Example: Constrained Optimization The nonlinear programming problem is to find x that minimizes the objective function S(x) = 10 j=1 x j (c j + ln x ) j, x x 10 (the c i are constants) subject to the following set of constraints: x 1 + 2x 2 + 2x 3 + x 6 + x 10 2 = 0, x 4 + 2x 5 + x 6 + x 7 1 = 0, x 3 + x 7 + x 8 + 2x 9 + x 10 1 = 0, x j , j = 1,..., 10. The Cross-Entropy Method formathematical Programming p. 28/42

69 The best known solution in Hock & Schittkowski (Test Examples for Nonlinear Programming Code) was x = ( , , , , , , , , , ) with S(x ) = However, using genetic algorithms Michalewicz (Genetic Algorithm + Data Structures = Evolution Programs) finds a better solution: x = ( , , , , , , , , , ), with S(x ) = The Cross-Entropy Method formathematical Programming p. 29/42

70 Using the CE method we can find an even better solution (in less time): x = ( , , , , , , , , , ) and S(x ) = (using ε = 10 4 ). The Cross-Entropy Method formathematical Programming p. 30/42

71 How was it done? The first step is to reduce the search space by expressing x 1, x 4, x 8 in terms of the other seven variables: x 1 = 2 (2x 2 + 2x 3 + x 6 + x 10 ), x 4 = 1 (2x 5 + x 6 + x 7 ), x 8 = 1 (x 3 + x 7 + 2x 9 + x 10 ). Hence, we have reduced the original problem of ten variables to that of a function of seven variables, which are subject to the constraints x 2, x 3, x 5, x 6, x 7, x 9, x , and 2 (2x 2 + 2x 3 + x 6 + x 10 ) , etc. The Cross-Entropy Method formathematical Programming p. 31/42

72 Next step choose an appropriate rectangular trust region R that contains the optimal solution. Draw the samples from a truncated normal distribution (with independent components) on this space R. Reject the sample if the constraints are not satisfied (acceptance rejection method). The updating formulas remain the same as for untruncated normal sampling: update via the mean and variance of the elite samples. An alternative is to add a penalty term to S. The Cross-Entropy Method formathematical Programming p. 32/42

73 Cross-Entropy: Some Theory Let X 1,..., X N be a random sample from f( ; v t 1 ). Let γ t be the (1 ρ) quantile of the performance: ρ = P vt 1 (S(X) γ t ), with X f( ; v t 1 ). Consider now the rare event estimation problem where we estimate ρ via Importance Sampling: ˆρ = 1 N N i=1 I {S(Xi ) γ t } f(x i ; v t 1 ) g(x i ), with X 1,..., X N g( ). A zero variance estimator is g (x) := I {S(x) γ t }f(x;v t 1 ) ρ. Problem: g depends on the unknown ρ. The Cross-Entropy Method formathematical Programming p. 33/42

74 Idea: Update v t such that the distance between the densities g and f( ; v t ) is minimal. The Cross-Entropy Method formathematical Programming p. 34/42

75 Idea: Update v t such that the distance between the densities g and f( ; v t ) is minimal. The Kullback-Leibler or cross-entropy distance is defined as: D(g, h) = E g log g(x) h(x) = g(x) log g(x) dx g(x) log h(x) dx. The Cross-Entropy Method formathematical Programming p. 34/42

76 Idea: Update v t such that the distance between the densities g and f( ; v t ) is minimal. The Kullback-Leibler or cross-entropy distance is defined as: D(g, h) = E g log g(x) h(x) = g(x) log g(x) dx g(x) log h(x) dx. Determine the optimal v t from min v D(g, f( ; v)). The Cross-Entropy Method formathematical Programming p. 34/42

77 Idea: Update v t such that the distance between the densities g and f( ; v t ) is minimal. The Kullback-Leibler or cross-entropy distance is defined as: D(g, h) = E g log g(x) h(x) = g(x) log g(x) dx g(x) log h(x) dx. Determine the optimal v t from min v D(g, f( ; v)). This is equivalent to solving max v E vt 1 I {S(X) γt } log f(x; v). The Cross-Entropy Method formathematical Programming p. 34/42

78 We may estimate the optimal solution v t by solving the following stochastic counterpart: max v 1 N N i=1 I {S(Xi ) γ t } log f(x i ; v), where X 1,..., X N is a random sample from f( ; v t 1 ). Alternatively, solve: 1 N N i=1 I {S(Xi ) γ t } log f(x i ; v) = 0, where the gradient is with respect to v. The Cross-Entropy Method formathematical Programming p. 35/42

79 The solution to the CE program can often be calculated analytically (Maximum Likelihood Estimator!). The Cross-Entropy Method formathematical Programming p. 36/42

80 The solution to the CE program can often be calculated analytically (Maximum Likelihood Estimator!). Note that the CE program is useful only when under v t 1 the event {S(X) γ t } is not too rare, say ρ The Cross-Entropy Method formathematical Programming p. 36/42

81 The solution to the CE program can often be calculated analytically (Maximum Likelihood Estimator!). Note that the CE program is useful only when under v t 1 the event {S(X) γ t } is not too rare, say ρ The Cross-Entropy Method formathematical Programming p. 36/42

82 The solution to the CE program can often be calculated analytically (Maximum Likelihood Estimator!). Note that the CE program is useful only when under v t 1 the event {S(X) γ t } is not too rare, say ρ The Cross-Entropy Method formathematical Programming p. 36/42

83 The solution to the CE program can often be calculated analytically (Maximum Likelihood Estimator!). Note that the CE program is useful only when under v t 1 the event {S(X) γ t } is not too rare, say ρ By iterating over t we obtain a sequence of reference parameters {v t, t 0} v and a sequence of levels {γ t, t 1} γ. The Cross-Entropy Method formathematical Programming p. 36/42

84 Updating Example Suppose X has iid exponential components: ( ) 5 5 x j f(x; v) = exp v j j=1 j=1 1 v j. The optimal v follows from the system of equations N i=1 I {S(Xi ) γ} W (X i ; u, w) log f(x i ; v) = 0. Since v j log f(x; v) = x j v 2 j 1 v j, The Cross-Entropy Method formathematical Programming p. 37/42

85 we have for the jth equation N i=1 I {S(Xi ) γ} ( Xij v 2 j 1 v j ) = 0, whence, v j = N i=1 I {S(X i ) γ}x ij N i=1 I {S(X i ) γ}, Thus the updating rule is here to take the mean of the elite samples (MLE). The Cross-Entropy Method formathematical Programming p. 38/42

86 Conclusions The CE method optimizes the sampling distribution. Advantages: Universally applicable (discrete/continuous/mixed problems) Very easy to implement Easy to adapt, e.g. when constraints are added Works generally well Disadvantages: It is a generic method Performance function should be relatively cheap Tweaking (modifications) may be required The Cross-Entropy Method formathematical Programming p. 39/42

87 Discussion + Further Research Synergies with related fields need to be further developed: Evolutionary Algorithms Stochastic Approximation Simulated Annealing Probability Collectives Reinforcement Learning Bayesian Inference (Multi-actor) game theory The Cross-Entropy Method formathematical Programming p. 40/42

88 Discussion + Further Research Convergence properties need to be further examined: Continuous optimization (e.g., Thomas Taimre s thesis) Rare event simulation (Homem-de-Melo, Margolin) Significance of various modifications Role of the sampling distribution Noisy optimization: very little needs to be changed! Test problems and best code need to be made available: CE Toolbox: The Cross-Entropy Method formathematical Programming p. 41/42

89 Acknowledgements Zdravko Botev, Jenny Liu, Sho Nariai, Thomas Taimre Phil Pollett The Cross-Entropy Method formathematical Programming p. 42/42

90 Acknowledgements Zdravko Botev, Jenny Liu, Sho Nariai, Thomas Taimre Phil Pollett Thank You! The Cross-Entropy Method formathematical Programming p. 42/42

PARALLEL CROSS-ENTROPY OPTIMIZATION. Dirk P. Kroese. Department of Mathematics University of Queensland Brisbane, QLD 4072, AUSTRALIA

Proceedings of the 27 Winter Simulation Conference S. G. Henderson, B. Biller, M.-H. Hsieh, J. Shortle, J. D. Tew, and R. R. Barton, eds. PARALLEL CROSS-ENTROPY OPTIMIZATION Gareth E. Evans Department