Solving Optimization Problems with MATLAB Loren Shure 6 The MathWorks, Inc.
Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization
Optimization Problems Maimize Fuel Efficiency Minimize Risk Maimize Profits
Design Process Modify Design Variables No Initial Design Variables System Objectives met? Yes Optimal Design 4
Why use Optimization? Manually (trial-and-error or iteratively) Initial Guess 5
Why use Optimization? Automatically (using optimization techniques) Initial Guess 6
Why use Optimization? Finding better (optimal) designs and decisions Faster design and decision evaluations Automate routine decisions Useful for trade-off analysis Antenna Design Using Genetic Algorithm http://ic.arc.nasa.gov/projects/esg/research/antenna.htm Non-intuitive designs may be found 7
Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization 8
Curve Fitting Demo Given some data: t = [..8..6.]; y = [.8.7.6.6.55.5]; Fit a curve of the form: y( t) c c e t 9
How to solve? Ec c c e y e c c t y t t ) ( As a linear system of equations: min y Ec Ec y c Can t solve this eactly (6 eqns, unknowns) An optimization problem!..6..8..5.55.6.6.7.8 c c e e e e e e
Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization
Nonlinear Optimization min log + 4 + +
Nonlinear Optimization - Modeling Gantry Crane Determine acceleration profile that minimizes payload swing Constraints : t f t s t s t 4s t p p p f t p s s 5s
Symbolic Math Toolbo Functions for analytical computations Integration Solving equations Simplification Differentiation Transforms Conveniently manage & document symbolic computations in Live Editor Math notation, embedded tet, graphics Share work as pdf or html Integrate with numeric computing MATLAB, Simulink and Simscape language Perform Variable-precision arithmetic Perform eact computations using familiar MATLAB synta in MATLAB 4
Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization 5
Mied-Integer Programming Many things eist in discrete amounts: Shares of stock Number of cars a factory produces Number of cows on a farm Often have binary decisions: On/off Buy/don t buy Mied-integer linear programming: Solve optimization problem while enforcing that certain variables need to be integer 6
Mied-Integer Linear Programming Continuous and integer variables, {,,,4,5} Linear objective and constraints such that min + 4 + = 7
Traveling Salesman Problem Problem How to find the shortest path through a series of points? Solution Calculate distances between all combinations of points Solve an optimization problem where variables correspond to trips between two points 8
Topics Introduction Least-squares minimization Nonlinear optimization Mied-integer programming Global optimization 9
Eample Global Optimization Problems Why does fmincon have a hard time finding the function minimum? Starting at Starting at Starting at sin() + cos( ) sin() + cos( ) 5-5 - 5 Starting at 6 5-5 - 5 5-5 - 5 Starting at 8 5-5 - 5 5-5 - 5 Starting at 5-5 - 5
Eample Global Optimization Problems Why didn t fminunc find the maimum efficiency? Initial Guess
Eample Global Optimization Problems Why didn t nonlinear regression find a good fit?.8 c=b e -b 4 t +b e -b 5 t +b e -b 6 t.7.6.5 c.4... 4 6 8 t
Global Optimization Goal: Want to find the lowest/largest value of the nonlinear function that has many local minima/maima Global Minimum at [ ] Problem: Traditional solvers often return one of the local minima (not the global) Solution: A solver that locates globally optimal solutions Rastrigin s Function
Global Optimization Solvers Covered Today Multi Start Global Search Simulated Annealing Pattern Search Particle Swarm Genetic Algorithm 4
MultiStart Demo Nonlinear Regression lsqcurvefit solution MultiStart solution 5
MULTISTART 6
What is MultiStart? Run a local solver from each set of start points Option to filter starting points based on feasibility Supports parallel computing 7
MultiStart Demo Peaks Function 8
GLOBAL SEARCH 9
What is GlobalSearch? Multistart heuristic algorithm Calls fmincon from multiple start points to try and find a global minimum Filters/removes non-promising start points
GlobalSearch Overview Schematic Problem Peaks function Three minima Green, z = -.65 Red, z= -.5 Blue, z = -6.55 y - - - -
GlobalSearch Overview Stage Run from specified y - - - -
GlobalSearch Overview Stage 4 6 y - - - -
GlobalSearch Overview Stage 4 6 y - - - - 4
GlobalSearch Overview Stage y - - - - 5
GlobalSearch Overview Stage y - - - - 6
GlobalSearch Overview Stage y - - - - 7
GlobalSearch Overview Stage y - - - - 8
GlobalSearch Overview Stage y - - - - 9
GlobalSearch Overview Stage 6 Current penalty threshold value : 4 y - - - - 4
GlobalSearch Overview Stage y - - - - 4
GlobalSearch Overview Stage Current penalty threshold value : 4 y - - - - - 4
GlobalSearch Overview Stage Epand basin of attraction if minimum already found Current penalty threshold value : y Basins can overlap - - - - -. 4
GlobalSearch Demo Peaks Function 44
SIMULATED ANNEALING 45
What is Simulated Annealing? A probabilistic metaheuristic approach based upon the physical process of annealing in metallurgy. Controlled cooling of a metal allows atoms to realign from a random higher energy state to an ordered crystalline (globally) lower energy state 46
Simulated Annealing Overview Iteration Run from specified y.9 - - - - 47
Simulated Annealing Overview Iteration Temperature = y.9 - Possible New Points: Standard Normal N(,) * Temperature - - - 48
Simulated Annealing Overview Iteration Temperature = P e accept (.9)/ T. y.9 - - - - 49
Simulated Annealing Overview Iteration Temperature = y.9 -. - - - 5
Simulated Annealing Overview Iteration Temperature = y.9 -. - - - 5
Simulated Annealing Overview Iteration Temperature = y.9 -. - - - 5
Simulated Annealing Overview Iteration Temperature =.75 y.9 -. -. - - - 5
Simulated Annealing Overview Iteration N- Temperature =. - y.9 -. -. - - - 54
Simulated Annealing Overview Iteration N Reannealing Temperature = y.9 -. - -. - - - 55
Simulated Annealing Overview Iteration N Reannealing Temperature = y.9 -. Paccept ( )) / T e - -. (.7 - - - 56
Simulated Annealing Overview Iteration N Reannealing Temperature = y.9 -. - -. - - - 57
Simulated Annealing Overview Iteration N+ Temperature =.75 y.9 -. - -. - - - - 58
Simulated Annealing Overview Iteration N+ Temperature =.75 y.9 -. - -. - - - - 59
Simulated Annealing Overview Iteration Temperature =.75 y.9 -. - -. - -6.5 - - - 6
Simulated Annealing Peaks Function 6
PATTERN SEARCH (DIRECT SEARCH) 6
What is Pattern Search? An approach that uses a pattern of search directions around the eisting points Epands/contracts around the current point when a solution is not found Does not rely on gradients: works on smooth and nonsmooth problems 6
Pattern Search Overview Iteration Run from specified y - - - - 64
Pattern Search Overview Iteration Apply pattern vector, poll new points for improvement Mesh size = Pattern vectors = [,], [,], [-,], [,-] First poll successful 4.6 Pnew mesh _ size* pattern _ vector.8.6 *[,] y.4 - Complete Poll (not default) - - - 65
Pattern Search Overview Iteration Mesh size = Pattern vectors = [,], [,], [-,], [,-] 4.6.8.6 y.8.4. - - - - -4 Complete Poll 66
Pattern Search Overview Iteration Mesh size = 4 Pattern vectors = [,], [,], [-,], [,-] 4.6.8.6 y.8.4. - -4 - - - 67
Pattern Search Overview Iteration 4 Mesh size = 4*.5 = Pattern vectors = [,], [,], [-,], [,-] 4.6.8.6 y.8.4. - -4 - - - 68
Pattern Search Overview Iteration N Continue epansion/contraction until convergence 4.6.8.6 y.8.4. - -6.5-4 - - - 69
Pattern Search Peaks Function 7
Pattern Search Climbs Mount Washington 7
PARTICLE SWARM 7
What is Particle Swarm Optimization? A collection of particles move throughout the region Particles have velocity and are affected by the other particles in the swarm Does not rely on gradients: works on smooth and nonsmooth problems 7
Particle Swarm Overview Iteration Initialize particle locations and velocities, evaluate all locations y - - - - 74
Particle Swarm Overview Iteration N Update velocities for each particle Best Location for this Particle y - Previous Velocity Best Location for Neighbor Particles - - - 75
Particle Swarm Overview Iteration N Update velocities for each particle y - New Velocity - - - 76
Particle Swarm Overview Iteration N Move particles based on new velocities y - - - - 77
Particle Swarm Overview Iteration N Continue swarming until convergence y - - - - 78
Particle Swarm Peaks Function 79
GENETIC ALGORITHM 8
What is a Genetic Algorithm? Uses concepts from evolutionary biology Start with an initial generation of candidate solutions that are tested against the objective function Subsequent generations evolve from the st through selection, crossover and mutation 8
How Evolution Works Binary Case Selection Retain the best performing bit strings from one generation to the net. Favor these for reproduction parent = [ ] parent = [ ] Crossover parent = [ ] parent = [ ] child = [ ] Mutation parent = [ ] child = [ ] 8
Genetic Algorithm Iteration Evaluate initial population y - - - - 8
Genetic Algorithm Iteration Select a few good solutions for reproduction y - - - - 84
Genetic Algorithm Iteration Generate new population and evaluate y - - - - 85
Genetic Algorithm Iteration y - - - - 86
Genetic Algorithm Iteration y - - - - 87
Genetic Algorithm Iteration y - - - - 88
Genetic Algorithm Iteration N Continue process until stopping criteria are met y - Solution found - - - 89
Genetic Algorithm Peaks Function 9
Genetic Algorithm Integer Constraints Mied Integer Optimization min f ( ) s.t. some are integers Eamples Only certain sizes of components available Can only purchase whole shares of stock 9
Application: Circuit Component Selection 6 components to size Only certain sizes available Objective: Match Voltage vs. Temperature curve Thermistor Circuit Ω Ω 6 Ω 8k Ω k Ω k Ω? Thermistors: Resistance varies nonlinearly with temperature R TH = R TH,Nom e βt T Nom T T Nom 9
Global Optimization Toolbo Solvers GlobalSearch, MultiStart Well suited for smooth objective and constraints Return the location of local and global minima ga, gamultiobj, simulannealbnd, particleswarm Many function evaluations to sample the search space Work on both smooth and nonsmooth problems patternsearch Fewer function evaluations than ga, simulannealbnd, particlewarm Does not rely on gradient calculation like GlobalSearch and MultiStart Works on both smooth and nonsmooth problems 9
Optimization Toolbo Solvers fmincon, fminbnd, fminunc, fgoalattain, fminima Nonlinear constraints and objectives Gradient-based methods for smooth objectives and constraints quadprog, linprog Linear constraints and quadratic or linear objective, respectively intlinprog Linear constraints and objective and integer variables lsqlin, lsqnonneg Constrained linear least squares lsqnonlin, lsqcurvefit Nonlinear least squares fsolve Nonlinear equations 94
Speeding-up with Parallel Computing Global Optimization solvers that support Parallel Computing: ga, gamultiobj: Members of population evaluated in parallel at each iteration patternsearch: Poll points evaluated in parallel at each iteration particleswarm: Population evaluated in parallel at each iteration MultiStart: Start points evaluated in parallel Optimization solvers that support Parallel Computing: fmincon: parallel evaluation of objective function for finite differences fminunc, fminima, fgoalattain,fsolve, lsqcurvefit, lsqnonlin: same as fmincon Parallel Computing can also be used in the Objective Function parfor 95
Parallel Computing Toolbo for the Desktop Desktop Computer Local Speed up parallel applications Take advantage of GPUs Prototype code for your cluster Simulink, Blocksets, and Other Toolboes MATLAB 96
Scale Up to Clusters and Clouds Desktop Computer Computer Cluster Local Cluster Simulink, Blocksets, and Other Toolboes MATLAB Scheduler 97
Learn More about Optimization with MATLAB Recorded webinar: Mied Integer Linear Programming in MATLAB MATLAB Digest: Using Symbolic Gradients for Optimization MATLAB Digest: Improving Optimization Performance with Parallel Computing Recorded webinar: Optimization in MATLAB for Financial Applications Cash Flow Matching Eample Recorded webinar: Optimization in MATLAB: An Introduction to Quadratic Programming Optimization Toolbo Web demo: Finding an Optimal Path using MATLAB and Optimization Toolbo 5 # Purchased 5 5 4 5 Bonds 98
Key Takeaways Solve a wide variety of optimization problems in MATLAB Linear and Nonlinear Continuous and mied-integer Smooth and Nonsmooth Find better solutions to multiple minima and non-smooth problems using global optimization Use symbolic math for setting up problems and automatically calculating gradients Using parallel computing to speed up optimization problems 99
Questions? 6 The MathWorks, Inc.