Optimization with Scilab

Transcription

1 Optimization with Scilab June 29 th 2011 Michaël BAUDIN & Vincent COUVERT Scilab Consortium

2 Outline Part 1 - What's new in Scilab 5? Focus on the Nelder-Mead component Part 2 - Part 3 - Part 4 - Optimization in Scilab: Matlab compatibility OMD2 project: Scilab Platform Development Conclusion What is missing in Scilab? The free and software open for source numerical software computation for numerical computation

3 Part 1 What's new in Scilab 5? 1. Introduction 1.1 What's in Scilab? 1.2 What's new in Scilab v5? 1.3 What's new on Atoms? 2. The Nelder-Mead Component 2.1 Introduction 2.2 The algorithm 2.3 Test cases 2.4 Conclusions The free and software open for source numerical software computation for numerical computation

4 1.1 What's in Scilab? Objective Bounds Equality Inequalities Size Gradient Needed Solver Linear yes linear linear medium - linpro Quadradic yes linear linear medium - quapro Quadratic yes linear linear large - qpsolve Quadratic yes linear linear medium - qld Non-Linear yes large yes optim Non-Linear small no fminsearch Non-Linear yes small no neldermead Non-Linear yes small no optim_ga Non-Linear small no optim_sa N.Li.Lea.Sq. large optional lsqrsolve N.Li.Lea.Sq. large optional leastsq Min-Max yes medium yes optim/nd Multi-Obj yes small no optim_moga Semi-Def. lin. (spectral) large no semidef L.M.I. lin. (spectral) lin. (spectral) large no lmisolve

5 1.2 What's new in Scilab 5? Genetic Algorithms: nonlinear objective, bounds, global optimization Simulated Annealing: nonlinear objective, global optimization The Nelder-Mead component: nonlinear objective, unconstrained, derivative-free, local optimization fminsearch: Matlab compatible

6 1.3 What's new on ATOMS? Optimization Solvers: Quapro: linear or quadratic objective, linear constraints (full matrices), SciIpopt: an interface to Ipopt. Nonlinear objective, nonlinear constraints (beta version), Fmincon: nonlinear objective, nonlinear constraints (alpha version) Matlab compatible, Other modules: Cobyla, Particle Swarm Optimization, Optkelley, Test Problems: Uncprb: 35 unconstrained optimization problems, AMPL: load AMPL problems into Scilab, And also: CUTEr.

7 Outline 1. Introduction 1.1 What's in Scilab? 1.2 What's new in Scilab v5? 1.3 What's new on Atoms? 2. The Nelder-Mead Component 2.1 Introduction 2.2 The algorithm 2.3 Test cases 2.4 Conclusions The free and software open for source numerical software computation for numerical computation

8 2. The Nelder-Mead Component 2.1 Introduction John Ashworth Nelder (8 October August 2010) Source:

9 2. The Nelder-Mead Algorithm 2.1 Introduction We are interested in solving the unconstrained continuous optimization problem: Minimize f(x) with unbounded, real, multidimensional, variable x. A direct search algorithm: Uses only function values (no gradient needed), Does not approximate the gradient. «A simplex method for function minimization», John Nelder, Roger Mead, Computer Journal, vol. 7, no 4, 1965, p

10 2. The Nelder-Mead Algorithm 2.1 Introduction Virginia Torczon (1989) writes: "Margaret Wright has stated that over fifty percent of the calls received by the support group for the NAG software library concerned the version of the Nelder-Mead simplex algorithm to be found in that library."

11 2. The Nelder-Mead Algorithm 2.2 The algorithm A simplex: a set of n+1 vertices, in n dimensions. In 2 dimensions. In 3 dimensions.

12 2. The Nelder-Mead Algorithm 2.2 The algorithm Steps in the Nelder-Mead algorithm Inputs: the n+1 vertices v(1), v(2),..., v(n+1) of a nondegenerate simplex in n dimensions, the associated function values f(1),...,f(n+1), the coefficients ρ (reflection), χ (expansion), γ (contraction), and σ (shrinkage). Standard Nelder-Mead: ρ=1, χ=2, γ=1/2, and σ=1/2.

13 2. The Nelder-Mead Algorithm 2.2 The algorithm

14 2. The Nelder-Mead Algorithm 2.3 Test cases f x 1, x 2 =x 1 2 x 2 2 x 1 x 2 function [ y, index ] = quadratic ( x, index ) y = x(1)^2 + x(2)^2 - x(1) * x(2); endfunction nm = neldermead_new (); nm = neldermead_configure(nm,"-numberofvariables",2); nm = neldermead_configure(nm,"-function",quadratic); nm = neldermead_configure(nm,"-x0",[2 2]'); nm = neldermead_search(nm); xopt = neldermead_get(nm,"-xopt"); nm = neldermead_destroy(nm);

15 2. The Nelder-Mead Algorithm 2.3 Test cases

16 2. The Nelder-Mead Algorithm 2.3 Test cases Mc Kinnon, «Convergence of the neldermead simplex method to a nonstationary point». SIAM J. on Optimization, 1998 Failure by repeated inside contraction

17 2. The Nelder-Mead Algorithm 2.3 Test cases C. T. Kelley. «Detection and remediation of stagnation in the neldermead algorithm using a sufficient decrease condition» SIAM J. on Optimization, 1999 Restart the algorithm...

18 2. The Nelder-Mead Algorithm 2.4 Conclusions Some general facts: Memory requirement is O(n²) Shrink steps are rare Generally 1 or 2 function evaluations by iteration Convergence is slow. Typical number of iterations is 100n, where n is the number of dimensions Hundreds of iterations are not rare Convergence can be even slower when n > 10 (Han & Neumann, 2006) Restart the algorithm when in doubt for convergence (Kelley, 1999) Convergence is guaranteed in 1 dimension (Lagarias et al., 1999)

19 2. The Nelder-Mead Algorithm 2.4 Conclusions We should not use this algorithm just because the gradient is not required: For example, if f is smooth, Quasi-Newton methods (optim) with numerical derivatives converge much faster. We may use this algorithm when: No other property of the problem can be used (e.g. non linear least squares can be solved by lsqrsolve), The objective function is nonsmooth or "noisy" (Kelley, 1999), We do not need too much accuracy (Torzcon, 1989), The number of parameters is moderate (Han & Neumann, 2006).

20 Part 3 Optimization in Scilab: Matlab compatibility 1. Introduction 2. Scilab Coverage 3. Overview The free and software open for source numerical software computation for numerical computation

21 1. Introduction Matlab has many functions for optimization: Minimization, Equation solving, Datafitting and nonlinear least squares, Global optimization. Scilab has often similar functions: let's see which ones. Matlab is a registered trademark of The Mathworks, Inc.

22 1. Introduction For each Matlab function, we search: Scilab function, if available, Differences of features, differences of algorithms. (*) : Function will be reviewed at the end of the talk, For most functions, the match is not 100% identical, But some other functions can do it : which ones? We consider only Scilab Industrial Grade solvers: Scilab internal modules, ATOMS modules, Portables on all OS, Well documented, Tested.

23 1. Introduction Main differences Design: Matlab : problem oriented (may be with several solvers), Scilab: solver oriented (may be several solvers). Function arguments: Matlab nearly always provides common options, Scilab is less homogeneous. Management of the callbacks/extra-arguments: Matlab : M-file Scilab: list

24 2. Scilab Coverage Minimization: fminbnd Not 100% identical, But optim can do it. fmincon ATOMS/fmincon (alpha version) fminimax Not 100% identical, but optim/''nd'' is designed for it. fminsearch fminsearch 90% identical in Scilab fminsearch 99% identical in 5.4.0

25 2. Scilab Coverage fminunc Not 100% identical, but optim/''qn'' or optim/''gc'' are designed for it. No sparsity pattern of Hessian in Scilab. No PCG in optim: L-BFGS instead. linprog 100% for full matrices: karmarkar ATOMS/quapro: linpro No known solver for sparse matrices (*). quadprog 100% for full matrices: qpsolve, qp_solve ATOMS/quapro: quapro No known solver for sparse matrices.

26 2. Scilab Coverage Equation Solving: fsolve fsolve 100% for full matrices. No known solver with sparse Jacobian (*). fzero No identical function. But fsolve can do it. Least Squares (Curve Fitting): lsqcurvefit datafit lsqnonlin lsqrsolve (leastsq)

27 2. Scilab Coverage Global Optimization Toolbox: Genetic Algorithm Not 100% identical, But optim_ga is built-in Scilab. No linear equality and inequality in Scilab, but bounds are managed. Simulated Annealing Not 100% identical, But optim_sa is built-in Scilab No bounds in Scilab SA, but user can customize the neighbour function.

28 3. Overview Matlab Problem Scilab bintprog Binary Integer Programming - fgoalattain Multiobjective goal attainment - fminbd Single-variable, on interval optim fmincon Constrained, nonlinear, multivariable ATOMS/fmincon fminimax Minimax, constrained optim/''nd'' fminsearch Unconstrained, multivariable, derivative-free fminsearch (100%) fminunc Unconstrained, multivariable optim/''qn'',''gc'' fseminf ktrlink Semi-infinitely constrained, multivariable, nonlinear Constrained or unconstrained, nonlinear, multivariable using Knitro linprog Linear programming karmarkar, ATOMS/quapro quadprog Quadratic programming qpsolve, ATOMS/quapro - -

29 3. Overview Matlab Problem Scilab fsolve Solve systems of nonlinear equations fsolve fzero Root of continuous function of one variable - lsqcurvefi t Nonlinear least squares curve fitting datafit lsqlin Constrained linear least squares - lsqnonlin Nonlinear least-squares (nonlinear data-fitting) lsqrsolve, leastsq lsqnonneg Nonnegative least squares - optimtool GUI to select solvers, options and run problems - Global Search Solve GlobalSearch problems - Multi Start Solve MultiStart problems - Genetic Algorithm Genetic Algorithms optim_ga Direct Search Pattern Search - Simulated Annealing Simulated Annealing optim_sa

30 Part 4 - OMD2 project: Scilab Platform Development 1. Overview 2. Modules 2.1 Data Management 2.2 Modeling 2.3 Optimization The free and software open for source numerical software computation for numerical computation

31 Overview OMD2 / CSDL projects collaboration Will be available on Scilab forge: Private project up to first release. Scilab Optimization Platform: Batch mode (script edition, large scale execution), GUI mode (interactive edition, prototyping). Future Scilab external module available through ATOMS.

32 Main functionalities Project management (Save & Load working data as HDF5 files) Wrappers: Scilab algorithms, External tools, Proactive. Mask complexity for users Modules: Data Management, Modeling, Optimization, Visualization.

33 Data Management Module (1/2) Factors / Parameters: Load existing Design Of Experiments (Isight.db files, ) Generate Design Of Experiments: DoE generator wrappers (LHS, ), DoE generator settings. Responses simulation using: External tool (openfoam, Catia, CCM+, ), Scilab function. 2-D visualization: Factor / Factor, Response / Factor.

34 Data Management Module (2/2)

35 Modeling module (1/2) Point selection: Learning points used for modeling, Validation points used to validate model, Bad points (simulation issue, ). Modeler: Selected among modeler wrappers (DACE, Lolimot, ), Parameters configuration, Multiple model management with best model user selection. Visualization: 2-D models, Cross correlation, Sensibility analysis.

36 Modeling module (2/2)

37 Optimization Module (1/2) Responses coefficients values setting Optimizer: Selection among generic wrappers (optim, fmincon, genetic algorithms, ), Optimizer configuration, Enable two chained optimizers. Visualization: Optimal point, Paretos, Robustness.

38 Optimization Module (2/2)

39 Part 4 - Conclusion 1. What is missing? 2. Bibliography The free and software open for source numerical software computation for numerical computation

40 Conclusion 1. What is missing? High Performance Optimization: Use BLAS/LAPACK within optim? Sparse Linear Programming: Update LIPSOL? Non Linear Programming: Improve fmincon? Non Linear Programming Test Cases: CUTEr requires a compiler on the test machine, Connect the Hock-Schittkowski collection?

41 Conclusion 2. Bibliography «Nelder-Mead User's Manual», Michaël Baudin, Consortium Scilab DIGITEO, 2010 «Optimization in Scilab», Baudin, Couvert, Steer, Consortium Scilab - DIGITEO INRIA, 2010 «Optimization with scilab, present and future», Michaël Baudin and Serge Steer, 2009 IEEE International Workshop on Open Source Software for Scientific Computation, pp , Sept «Introduction to Optimization with Scilab», Michaël Baudin, Consortium Scilab DIGITEO, 2010 «Unconstrained Optimality Conditions with Scilab», Michaël Baudin, Consortium Scilab DIGITEO, 2010

42 Thanks for your attention

43 Extra-Slides Some slides you won't see, unless you ask...

44 The Nelder-Mead Algorithm Some Historical References: Spendley, Hext, Himsworth (1962): fixed shape simplex algorithm Nelder, Mead (1965): variable shape algorithm Box (1965): simplex algo., with constraints O'Neill (1971): Fortran 77 implementation. Torczon (1989): Multi Directional Search. Mc Kinnon (1998): Counter examples of N-M. Lagarias, Reeds, Wright, Wright (1998): Proof of convergence in dimensions 1 and 2 for strictly convex functions. Han, Neumann (2006): More counter examples of N-M.

45 The Nelder-Mead Algorithm In what softwares N-M can be found? Matlab (fminsearch) NAG (E04CBF) Numerical Recipes (amoeba) IMSL (UMPOL) and Scilab since v5.2.0 in 2009 and R after Sébastien Bihorel's port of Scilab's source code.

46 The Nelder-Mead Algorithm 1. Sort by function value. Order the vertices: f(1) f(n) f(n+1) 2. Calculate centroid. B = (v(1)+...+v(n))/n 3. Reflection. Compute R = (1+ρ)B ρv(n+1) and evaluate f(r). 4. Expansion. If f(r)<f(1), compute E=(1+ρχ)B ρχv(n+1) and evaluate f(e). If f(e)<f(r), accept E, else accept R and goto Accept R. If f(1) f(r) < f(n), accept R and goto Outside Contraction. If f(n) f(r)<f(n+1), compute Co=(1+ργ)B ργv(n+1) and evaluate f(co). If f(co)<f(r), then accept Co and goto 1 else, goto Inside Contraction. If f(n+1) f(r), compute Ci=(1-γ)B +γv(n+1) and evaluate f(ci). If f(ci)<f(n+1), then accept Ci and goto 1 else, goto Shrink. Compute the points v(i)=v(1)+σ(v(i)-v(1)) and evaluate f(i)=f(x(i)), for i=2,3,...,n+1. Goto 1.

47 The Nelder-Mead Algorithm Lagarias, Reeds, Wright, Wright (1998) 1. In dimension 1, the Nelder-Mead method converges to a minimizer, and convergence is eventually M-step linear, when the reflection parameter ρ = In dimension 2, the function values at all simplex vertices in the standard Nelder-Mead algorithm converge to the same value. 3. In dimension 2, the simplices in the standard Nelder-Mead algorithm have diameters converging to zero. Note that Result 3 does not implies that the simplices converge to a single point x*.

48 What's in Matlab? Minimization: bintprog Solve binary integer programming problems fgoalattain Solve multiobjective goal attainment problems fminbnd Find minimum of single-variable function on fixed interval fmincon Find minimum of constrained nonlinear multivariable function fminimax Solve minimax constraint problem fminsearch Find minimum of unconstrained multivariable function using derivative-free method

49 What's in Matlab? fminunc Find minimum of unconstrained multivariable function fseminf Find minimum of semi-infinitely constrained multivariable nonlinear function ktrlink Find minimum of constrained or unconstrained nonlinear multivariable function using KNITRO third-party libraries linprog Solve linear programming problems quadprog Quadratic programming

50 What's in Matlab? Equation Solving: fsolve Solve system of nonlinear equations fzero Find root of continuous function of one variable Least Squares (Curve Fitting): lsqcurvefit Solve nonlinear curve-fitting (data-fitting) problems in least-squares sense lsqlin Solve constrained linear least-squares problems lsqnonlin Solve nonlinear least-squares problems lsqnonneg Solve nonnegative least-squares constraint problem

51 What's in Matlab? Utilities: optimtool GUI to select solver, optimization options, and run problems optimget Optimization options values optimset Create or edit optimization options structure

52 What's in Matlab toolboxes? Global Optimization Toolbox: GlobalSearch Create and solve GlobalSearch problems MultiStart Create and solve MultiStart problems Genetic Algorithm Use genetic algorithm and Optimization Tool, and modify genetic algorithm options Direct Search Use direct search and Optimization Tool, and modify pattern search options Simulated Annealing Use simulated annealing and Optimization Tool, and modify simulated annealing options