Topics. Introduction. Specific tuning/troubleshooting topics "It crashed" Gurobi parameters The tuning tool. LP tuning. MIP tuning

Transcription

1 Tuning II

2 Topics Introduction Gurobi parameters The tuning tool Specific tuning/troubleshooting topics "It crashed" The importance of being specific LP tuning The importance of scaling MIP tuning Performance variability Tuning improvements in 7.0 Model-building runtime Diagnosing Infeasibility

3 Gurobi Parameters

4 Gurobi parameters Version 7.0 has 117 parameters Can be grouped by function: Termination: control termination of the solvers Example: TimeLimit Tolerances: control solver tolerances Example: FeasibilityTol Simplex: affect the operation of the simplex algorithms Example: SimplexPricing Barrier: affect the operation of the barrier algorithm Example: Crossover MIP: affect the operation of the MIP algorithm Example: MIPFocus MIP Cuts: affect the MIP cutting planes Example: CutPasses Tuning: affect the operation of the automatic parameter tuner Example: TuneTimeLimit Distributed: affect the behavior of the distributed algorithms Example: WorkerPool Other

5 Gurobi parameters Of these 117, 57 are related to performance Multiple possible values for most We could tell you when each is appropriate Except that would take all day And we often don't know ourselves Better to use the Gurobi parameter tuning tool

6 The Gurobi Parameter Tuning Tool

7 Parameter tuning basics Parameter changes can have a big effect Parameter tuning tool captures simple rules of thumb "Always try primal, dual, and barrier for an LP" "Don't vary barrier parameters if you are using simplex" "If presolve consumes most of the runtime, try Presolve=1 Plus brute force More important than you might expect

8 grbtune: parameter tuning tool Automatically finds sets of parameters that give good performance Two criteria Primary: fastest time to optimality Secondary: Smallest MIP gap in a fixed amount of time is default Optional: best feasible, best bound Can run distributed across multiple computers An API is available

9 Tuning tool Tuning is harder than it may first appear We'll return to this topic later

10 Specific Troubleshooting/Tuning Topics

11 "It Crashed"

12 "It crashed" User definitions of "crash" (from actual support cases) Windows "blue screen" Segmentation fault (Linux, Mac) Fails to start optimization Solver returned no solution Binary variable took a value of Solution violated constraint by Solution objective was 0.002% above mathematical optimum No log entries for a long time Ran for a really long time

13 "It crashed" Resolution Windows "blue screen" Hardware problem: user program can't crash OS Segmentation fault (Linux, Mac) Problem in user program Fails to start optimization License issue Solver returned no solution Infeasible model, no status check Binary variable took a value of Tolerances (IntFeasTol) Solution violated constraint by Tolerances (FeasibilityTol) Solution objective was 0.002% above mathematical optimum Tolerances (MIPGap) No log entries for a long time Large and difficult model Ran for a really long time Large and difficult model Rarely: Bug in Gurobi libraries

14 "It crashed" Try to be specific Helps us figure out what's going on Often helps you to resolve the issue

15 Performance Variability and Tuning

16 Demonstration MIP tuning We use a model named acc-tight1.mps Apply tuning with TuneTrials=1 and TuneTimeLimit=30 Interactive shell commands: m = read( c:/users/*/desktop/acc-tight1.mps ) m.params.tunetrials = 1 m.params.tunetimelimit = 30 m.tune() Command Line Tool: grbtune TuneTrials=1 TuneTimeLimit=30 acc-tight1

17 Performance variability Output (grbtune TuneTrials=1 TuneTimeLimit=30 acc-tight1): Baseline parameter set: runtime 0.58s Improved parameter set 1 (runtime 0.20s): Method 0 MIPFocus 2 CutPasses 3 Improved parameter set 2 (runtime 0.22s): Method 0 MIPFocus 2

18 Tuning tips performance variability Good news! Model runs almost 3X faster with tuning First question does it make sense? Most important tuned parameter (Method=0) uses primal simplex for root solve Default Method setting: Root relaxation: objective e+00, 1617 iterations, 0.09 seconds Method=0: Root relaxation: objective e+00, 3204 iterations, 0.13 seconds Well that's strange

19 Tuning tips performance variability Second question: do benefits of parameter tuning survive small perturbations? Change random number seed (with Method=0 and MIPFocus=2): Seed=default: Seed=1: Seed=10: Seed=100: Seed=1000: Explored 0 nodes (3550 simplex iterations) in 0.22 seconds Explored 0 nodes (3168 simplex iterations) in 0.40 seconds Explored 0 nodes (3766 simplex iterations) in 1.90 seconds Explored 0 nodes (4061 simplex iterations) in 0.44 seconds Explored 0 nodes (3489 simplex iterations) in 0.29 seconds Performance very sensitive to small perturbations

20 Tuning tips performance variability A closer look Nodes Current Node Objective Bounds Work Expl Unexpl Obj Depth IntInf Incumbent BestBd Gap It/Node Time s s s s H % - 1s Optimization stops as soon as heuristic finds a solution Heuristics highly influenced by random effects Recipe for large runtime variations

21 Tuning tips performance variability That's why we perform multiple random trials What happens when we vary TuneTrials for this model? Variance is model dependent For model misc07: 3 trials: baseline runtime 0.40s, best runtime 0.25s 5 trials: baseline runtime 0.35s, best runtime 0.29s 10 trials: baseline runtime 0.32s, best runtime 0.30s 1 trial: baseline runtime 1.49s, best runtime 0.50s 3 trials: baseline runtime 1.44s, best runtime 0.51s 5 trials: baseline runtime 1.48s, best runtime 0.47s 10 trials: baseline runtime 1.51s, best runtime 0.44s Important to avoid over-tuning

22 Over-tuning

23 Tuning tips Tuning tool reports all potential improvements Even when improvement is very small For misc07 with TuneTrials=10, best reported parameter set: Improved parameter set 1 (runtime 0.45s): SimplexPricing 0 NormAdjust 2 Heuristics 0 VarBranch 1 Cuts 0 Possibilities to overtune are everywhere Parameter changes can produce random effects small benefits benefits that are specific to one instance, not your whole family of models

24 Tuning tips Tip #1: tune over multiple models grbtune misc05.mps misc06.mps misc07.mps Note that this will tune over the sum of the runtimes Models that take more time will have more influence Tip #2: less is more - pick the important parameters Tuning tool will report an efficient frontier Best results with 1 parameter change, 2 parameter changes, 3 Diminishing returns Stop when the marginal improvement gets small Tip #3: retune with each Gurobi release We work hard to make the default settings as effective as possible Changing a parameter takes you away from this default behavior

25 Tuning Improvements In 7.0

26 Tuning improvements A few tuning improvements in 7.0, based on user requests Can now tune with a MIP start Can now tune for specific criteria: Minimize gap (v6.5 behavior) Best incumbent objective Best lower bound

27 Scaling

28 Importance of scaling Scaling has a huge impact on performance A poorly scaled model can lead to Slow runtimes Feasibility violations In extreme cases, inability to find a feasible solution Signs of scaling trouble: Coefficient statistics: Matrix range [5e-05, 2e+04] Objective range [2e-03, 2e+09] Bounds range [4e-04, 4e+08] RHS range [3e+01, 9e+04] A few rules of thumb Spread between smallest and largest matrix coefficient < 1e6 (1e8 if you are feeling lucky) Largest bound and objective coefficient < 1e8

29 Importance of scaling Simple program to create a rescaled version of any model scaling = for var in model.getvars(): if var.vtype == GRB.CONTINUOUS: scale = random.uniform(scaling/2.0, scaling*2.0) flip = random.randint(0,3) if flip == 0: scale = 1.0 elif flip == 1: scale = 1.0/scale col = model.getcol(var) for i in range(col.size()): coeff = col.getcoeff(i) row = col.getconstr(i) model.chgcoeff(row, var, coeff*scale) var.obj = var.obj*scale if var.lb > -GRB.INFINITY: var.lb = var.lb/scale if var.ub < GRB.INFINITY: var.ub = var.ub/scale

30 Importance of scaling For LP model pds-20 with original scaling: With scaling=1e4: Coefficient statistics: Matrix range [1e+00, 1e+00] Objective range [3e+01, 1e+05] Bounds range [8e+00, 2e+04] RHS range [3e+01, 9e+04] Solved in iterations and 0.97 seconds Optimal objective e+10 Coefficient statistics: Matrix range [5e-05, 2e+04] Objective range [2e-03, 2e+09] Bounds range [4e-04, 4e+08] RHS range [3e+01, 9e+04] Solved in iterations and 2.15 seconds Optimal objective e+10

31 Importance of scaling With scaling=1e5: Coefficient statistics: Matrix range [5e-06, 2e+05] Objective range [2e-04, 2e+10] Bounds range [4e-05, 4e+09] RHS range [3e+01, 9e+04] Solved in iterations and 1.35 seconds Infeasible model

32 Importance of scaling Can parameter tuning fix this? Tuning tool results for model pds-20 with scaling=1e4: Improved parameter set 1 (runtime 1.80s): NormAdjust 1 PrePasses 8 Improved parameter set 2 (runtime 1.92s): NormAdjust 1 Aggressive scaling (ScaleFlag=2) does help (tuning tool won't find everything): Solved in iterations and 1.36 seconds Optimal objective e+10

33 Importance of scaling What about ScaleFlag=2 for scaling=1e5 model? Iteration Objective Primal Inf. Dual Inf. Time e e e+07 0s Warning: 1 variables dropped from basis e e e+09 5s e e e+10 11s e e e+10 16s e e e+09 20s e e e+10 26s e e e+09 31s e e e+18 56s e e e+18 60s e e e+09 65s e e e+00 65s Solved in iterations and seconds Optimal objective e+10 Warning: unscaled dual violation = e-05 and residual = e-06

34 Why is scaling so important? Computers use finite-precision floating-point arithmetic Trivial example: compute If results are computed to 4 digits: = =.1 To three digits, result is 0 When solving a linear system of equations, these round-off errors happen all over the place Potential for accumulated errors is captured in matrix condition number The larger the condition number, the less accurate the results

35 Importance of scaling Default Gurobi tolerances are ~1e-6 We need at least 6 accurate digits Otherwise, can't tell the difference between model feasibility problems and floating-point errors In modern computers, roughly 16 digits of accuracy in computed results IEEE 754 floating-point standard Simple interpretation of condition number: Multiply smallest floating-point error by condition number to get linear solve error Example: 2e-16 IEEE error * 1e8 condition number = 2e-8 linear solve error For model pds-20: scaling=1e2: condition number of optimal basis ~2e6 scaling=1e4: condition number of optimal basis ~1e10

36 Importance of scaling Why can't Gurobi recover the original scaling? We try, but it's a very difficult problem

37 Model Building Runtime

38 Time to build a model Fairly common type of support question " I set up a problem with +/ rows and columns in an internal data structure. Then I push this to Gurobi. The calls to GRBModel.addConstr( ) in total take 4 seconds. The call to optimize() later also takes 4 seconds. So it seems there is a lot of time lost by constructing Java wrappers to write the GRBModel to the underlying library." TL;DR: Why is the Gurobi API so slow?

39 Time to build a model Simple demonstration: Build a model with 100K constraints, 1M vars, and 1M non-zero coefficients: m = Model() for n in range( ): m.addvar(vtype=grb.binary) m.update() vars = m.getvars() for n in range(100000): expr = sum(vars[10*n:10*(n+1)]) m.addconstr(expr <= 1) On my laptop, above Python code takes 7.6s Python is our slowest API (Python is an interpreted language) C version is >10X faster No reason for a well-written program to spend significant time in Gurobi model building

40 Common model building pitfalls Common pitfall: too frequent calls to update() m = Model() for n in range( ): m.addvar(vtype=grb.binary) m.update() vars = m.getvars() for n in range(100000): expr = sum(vars[10*n:10*(n+1)]) m.addconstr(expr <= 1) m.update() Program now takes 1000+s (versus 7.6s) Only call update() when you need to look at the result of previous changes API change in Gurobi 7.0: most programs will never need to call update()

41 Common model building pitfalls Common pitfall: time not actually spent in Gurobi calls Simple approach to tracking this down Use Gurobi recording (Record parameter) Run your program with Record=1 Produces a recording000.grbr file Next, replay the recording file: gurobi_cl recording000.grbr Replays same sequence of Gurobi API calls exactly Produces runtime summary at the end *Replay* Gurobi API routine runtime: 2.12s *Replay* Gurobi solve routine runtime: 23.92s

42 Common model building pitfalls Common pitfall: not enough memory A model with 1M variables isn't going to solve on a machine with 4GB of memory Even if you remove all variable names Even if you store your coefficients in single precision BTW, never store your coefficients in single precision Surprising (to us) how frequently people run into this No simple rule to predict how much memory a model will require Best approach: Collect problems of different sizes Measure memory use versus problem size Use this to extrapolate to desired problem size

43 Infeasibility

44 Infeasibility detection tools Irreducible Inconsistent Subsystem (IIS) Finds a minimal set of constraints that conflict Attributes indicate what cannot be satisfied Useful for model development feasrelax Finds a solution with the minimum constraint violation Retrieve added artificial slack variables to find violations Useful for deployed application

45 More resources Sections in the Reference Manual Logging Parameter Guidelines Parameter Tuning Gurobi Examples Tuning example Infeasibility examples (particularly workforce4) Gurobi support: