SOLVING BUSINESS PROBLEMS WITH SAS ANALYTICS AND OPTMODEL

Transcription

1 SOLVING BUSINESS PROBLEMS WITH SAS ANALYTICS AND OPTMODEL INFORMS CONFERENCE ON BUSINESS ANALYTICS AND OPERATIONS RESEARCH HYATT REGENCY GRAND CYPRESS ORLANDO, FL TECHNOLOGY WORKSHOP APRIL 10, 2016

2 SAS ANALYTICS AND OPTMODEL WHAT WE LL COVER Background: SAS and SAS Analytics Introductory OPTMODEL Example: Political Redistricting Pharmaceutical Launch Revenue Optimization Product Formulation Optimization Pricing Optimization Technical Overview and Future Features

3 BACKGROUND: SAS AND SAS ANALYTICS ED HUGHES

4 SAS ANALYTICS AND OPTMODEL SAS BACKGROUND SAS Institute Inc. founded in 1976 SAS users, partners, and offices around the world Broad range of integrated analytic products Data access and integration Data mining Statistical analysis Forecasting and econometrics Text analytics Optimization, simulation, and scheduling

5 SAS ANALYTICS AND OPTMODEL OPTIMIZATION WITH SAS/OR OPTMODEL: algebraic optimization modeling language Linear Programming: Primal, dual, network simplex; interior point; concurrent Mixed Integer Linear Programming: Branch and cut: heuristics, conflict search, option tuning Decomposition algorithm (MILP and LP) Quadratic Programming: Interior point Nonlinear Programming: Active set; interior point; concurrent; multistart algorithm

6 SAS ANALYTICS AND OPTMODEL OPTIMIZATION WITH SAS/OR Network Algorithms 11 diagnostic and optimization algorithms Constraint Logic Programming Solve CSPs (general and scheduling) Local Search Hybrid parallel approach; multiple algorithms

7 SAS ANALYTICS AND OPTMODEL SAMPLE CUSTOMER PROJECTS: OPTIMIZATION Television advertising optimization (network) Chemical mixture optimization (Procter & Gamble) Optimal ATM replenishment (DBS Singapore) Public water management (national utilities agency) Optimal loan assignment (bank) Pharmaceutical launch revenue optimization

8 SAS ANALYTICS AND OPTMODEL RECENT CUSTOMER PROJECTS: SIMULATION Hospital room/bed assignment Neonatal intensive care unit (Duke Children s Hospital) Prison population planning (NC Sentencing Commission) Digital parts distribution (parts distributor/retailer)

9 SAS ANALYTICS AND OPTMODEL OPTIMIZATION-DRIVEN SOLUTIONS SAS Marketing Optimization SAS Size Optimization SAS Service Parts Optimization SAS for Service Operations Optimization SAS Risk Management for Banking, Insurance SAS Markdown Optimization SAS Credit Scoring for Banking SAS Customer Link Analytics SAS Fraud Framework SAS Inventory Optimization Workbench SAS Launch Revenue Optimization

10 IDC WORLDWIDE ADVANCED AND PREDICTIVE ANALYTICS SOFTWARE 2014 VENDOR SHARES Source: IDC, Worldwide Advanced and Predictive Analytics Software Market Shares, 2014: The Rise of the Long Tail (Doc # )., July 2015.

11 GARTNER: MAGIC QUADRANT FOR ADVANCED ANALYTIC PLATFORMS Gartner Magic Quadrant for Advanced Analytics Platforms by Lisa Kart, Gareth Herschel, Alexander Linden, Jim Hare 9 February This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from SAS. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

12 INTRODUCTORY OPTMODEL EXAMPLE: POLITICAL REDISTRICTING ROB PRATT

13 GRAPH PARTITIONING PROBLEM DESCRIPTION Given undirected graph, node weights, positive integer k, target t [l, u] Partition node set into k subsets, each with total weight within [l, u] Usual objective: minimize sum of edge weights between parts of partition Our objective: find most balanced partition Minimize total absolute deviation of subset weight from target (L 1 norm) Minimize maximum absolute deviation of subset weight from target (L norm) Connectivity constraints: each of the k resulting induced subgraphs must be connected

14 GRAPH PARTITIONING MOTIVATION Dennis E. Shasha, Doctor Ecco s Cyberpuzzles (2002) Political districting Advanced Analytics and Optimization Services consulting engagement Assigning service workers to customer regions INFORMS OR Exchange posting (Jan. 2013) Forestry

15 GRAPH PARTITIONING MAP FROM SHASHA (2002)

16 GRAPH PARTITIONING MAP REPRESENTED AS GRAPH

17 GRAPH PARTITIONING NODE WEIGHTS (POPULATIONS)

18 GRAPH PARTITIONING FEASIBLE SOLUTION FOR 14 CLUSTERS, BOUNDS ± 100

19 GRAPH PARTITIONING SOLUTION APPROACHES Compact MILP formulation with binary variables, nonnegative flow variables, flow balance constraints Row generation, dynamically excluding disconnected subsets Column generation, dynamically generating connected subsets Column generation, statically generating all possible connected subsets

20 GRAPH PARTITIONING PROC OPTMODEL CODE: MILP SOLVER Master MILP problem: Given all feasible subsets, solve cardinality constrained set partitioning problem 1. /* master problem: partition nodes into connected subsets, minimizing deviation from target */ 2. /* UseSubset[s] = 1 if subset s is used, 0 otherwise */ 3. var UseSubset {SUBSETS} binary; /* L_1 norm */ 6. num subset_population {s in SUBSETS} = sum {i in NODES_s[s]} p[i]; 7. num subset_deviation {s in SUBSETS} = abs(subset_population[s] - target); 8. min Objective1 = sum {s in SUBSETS} subset_deviation[s] * UseSubset[s]; /* each node belongs to exactly one subset */ 11. con Partition {i in NODES}: 12. sum {s in SUBSETS: i in NODES_s[s]} UseSubset[s] = 1; /* use exactly the allowed number of subsets */ 15. con Cardinality: 16. sum {s in SUBSETS} UseSubset[s] = num_clusters; problem Master include 19. UseSubset Objective1 Partition Cardinality;

21 GRAPH PARTITIONING STATIC COLUMN GENERATION 1. Generate all possible connected subsets with weights close enough to target Relax connectivity and find all possible subsets Postprocess to filter out disconnected subsets 2. Solve cardinality constrained set partitioning master problem

22 GRAPH PARTITIONING FEATURES USED IN PROC OPTMODEL MILP solver to compute maximum subset size for each node (COFOR loop) Network solver (all-pairs shortest path) to generate valid cuts that encourage connectivity Constraint programming solver to generate subsets (multiple solutions) Network solver (connected components) to check connectivity (COFOR loop) MILP solver to solve set partitioning problem

23 GRAPH PARTITIONING COLUMN GENERATION SUBPROBLEM Given node populations p i and target population range [l, u] Find all connected node subsets close to target population Binary variable x i indicates whether node i appears in subset Range constraint l i N p i x i u Connectivity hard to impose directly, so relax (but do not ignore)

24 GRAPH PARTITIONING NODE SUBSET 1, 2, 3, 4, 5 SATISFIES POPULATION BOUNDS, BUT IS DISCONNECTED

25 GRAPH PARTITIONING VALID CUTS FOR NODES TOO FAR APART m i : max # nodes among feasible subsets that contain node i If shortest path distance d ij min m i, m j then x i + x j 1

26 GRAPH PARTITIONING PROC OPTMODEL CODE: MILP SOLVER 1. /* maxnodes[i] = max number of nodes among feasible subsets that contain node i */ 2. num maxnodes {NODES}; /* UseNode[i] = 1 if node i is used, 0 otherwise */ 5. var UseNode {NODES} binary; max NumUsed = sum {i in NODES} UseNode[i]; con PopulationBounds: 10. target - deviation_ub <= sum {i in NODES} p[i] * UseNode[i] <= target + deviation_ub; problem MaxNumUsed include 13. UseNode NumUsed PopulationBounds; 14. use problem MaxNumUsed; put Finding max number of nodes among feasible subsets that contain node i... ; 17. cofor {i in NODES} do; 18. put i=; 19. fix UseNode[i] = 1; 20. solve with MILP / loglevel=0 logfreq=0; 21. maxnodes[i] = round(numused.sol); 22. unfix UseNode; 23. end;

27 GRAPH PARTITIONING PROC OPTMODEL CODE: NETWORK SOLVER 1. /* distance[i,j] = shortest path distance (number of edges) from i to j */ 2. num distance {NODES, NODES}; 3. /* num one {EDGES} = 1; */ 4. put Finding all-pairs shortest path distances... ; 5. solve with network / 6. links = (include=edges /*weight=one*/) 7. shortpath 8. out = (spweights=distance);

28 GRAPH PARTITIONING VALID CUTS FOR SMALL-POPULATION NODES If x i = 1 and p i < l then x j = 1 for some neighbor j of i x i jεni x j

29 GRAPH PARTITIONING VALID CUTS FOR SMALL-POPULATION SETS More generally, if x i = 1 for some i S N such that j S p j < l then x k = 1 for some neighbor k in moat around S x i kε( j N j) S x k Known as ring inequalities in the literature

30 GRAPH PARTITIONING PROC OPTMODEL CODE: CONSTRAINT PROGRAMMING SOLVER 1. set NODE_PAIRS = {i in NODES, j in NODES: i < j}; /* cannot have two nodes i and j that are too far apart */ 4. con Conflict {<i,j> in NODE_PAIRS: distance[i,j] >= min(maxnodes[i],maxnodes[j])}: 5. UseNode[i] + UseNode[j] <= 1; /* if UseNode[i] = 1 and p[i] < target - deviation_ub 8. then UseNode[j] = 1 for some neighbor j of i */ 9. con Moat1 {i in NODES: p[i] < target - deviation_ub}: 10. UseNode[i] <= sum {j in NEIGHBORS[i]} UseNode[j]; /* if (UseNode[i] = 1 or UseNode[j] = 1) and p[i] + p[j] < target - deviation_ub 13. then UseNode[k] = 1 for some neighbor k in moat around {i,j} */ 14. con Moat2_i {<i,j> in NODE_PAIRS: p[i] + p[j] < target - deviation_ub}: 15. UseNode[i] <= sum {k in (NEIGHBORS[i] union NEIGHBORS[j]) diff {i,j}} UseNode[k]; 16. con Moat2_j {<i,j> in NODE_PAIRS: p[i] + p[j] < target - deviation_ub}: 17. UseNode[j] <= sum {k in (NEIGHBORS[i] union NEIGHBORS[j]) diff {i,j}} UseNode[k]; problem Subproblem include 20. UseNode PopulationBounds Conflict Moat1 Moat2_i Moat2_j; 21. use problem Subproblem; put Solving CLP subproblem... ; 24. solve with CLP / findallsolns; 25. /* use _NSOL_ and.sol[s] to access multiple solutions */ 26. SUBSETS = 1.._NSOL_; 27. for {s in SUBSETS} NODES_s[s] = {i in NODES: UseNode[i].sol[s] > 0.5};

31 GRAPH PARTITIONING CONSTRAINT PROGRAMMING SOLVER LOG NOTE: Problem generation will use 4 threads. NOTE: The problem has 66 variables (0 free, 0 fixed). NOTE: The problem has 66 binary and 0 integer variables. NOTE: The problem has 5123 linear constraints (5122 LE, 0 EQ, 0 GE, 1 range). NOTE: The problem has linear constraint coefficients. NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range). NOTE: All possible solutions have been found. NOTE: Number of solutions found = CARD(SUBSETS)=4472

32 GRAPH PARTITIONING PROC OPTMODEL CODE: NETWORK SOLVER 1. /* check connectivity */ 2. set NODES_THIS; 3. num component {NODES_THIS}; 4. set DISCONNECTED init {}; 5. cofor {s in SUBSETS} do; 6. put s=; 7. NODES_THIS = NODES_s[s]; 8. solve with network / 9. links = (include=edges) 10. subgraph = (nodes=nodes_this) 11. concomp 12. out = (concomp=component); 13. if or {i in NODES_THIS} (component[i] > 1) 14. then DISCONNECTED = DISCONNECTED union {s}; 15. end; 16. SUBSETS = SUBSETS diff DISCONNECTED;

33 GRAPH PARTITIONING NETWORK SOLVER LOG s=1 NOTE: The SUBGRAPH= option filtered 145 elements from EDGES. NOTE: The number of nodes in the input graph is 4. NOTE: The number of links in the input graph is 3. NOTE: Processing connected components. NOTE: The graph has 1 connected component. NOTE: Processing connected components used 0.00 (cpu: 0.00) seconds.... s=28 NOTE: The SUBGRAPH= option filtered 144 elements from EDGES. NOTE: The number of nodes in the input graph is 6. NOTE: The number of links in the input graph is 4. NOTE: Processing connected components. NOTE: The graph has 2 connected components. NOTE: Processing connected components used 0.00 (cpu: 0.00) seconds.... s=4472 NOTE: The SUBGRAPH= option filtered 144 elements from EDGES. NOTE: The number of nodes in the input graph is 5. NOTE: The number of links in the input graph is 4. NOTE: Processing connected components. NOTE: The graph has 1 connected component. NOTE: Processing connected components used 0.00 (cpu: 0.00) seconds. CARD(SUBSETS)=2771

34 GRAPH PARTITIONING PROC OPTMODEL CODE: MILP SOLVER 1. /* master problem: partition nodes into connected subsets, minimizing deviation from target */ 2. /* UseSubset[s] = 1 if subset s is used, 0 otherwise */ 3. var UseSubset {SUBSETS} binary; /* L_1 norm */ 6. num subset_population {s in SUBSETS} = sum {i in NODES_s[s]} p[i]; 7. num subset_deviation {s in SUBSETS} = abs(subset_population[s] - target); 8. min Objective1 = sum {s in SUBSETS} subset_deviation[s] * UseSubset[s]; /* each node belongs to exactly one subset */ 11. con Partition {i in NODES}: 12. sum {s in SUBSETS: i in NODES_s[s]} UseSubset[s] = 1; /* use exactly the allowed number of subsets */ 15. con Cardinality: 16. sum {s in SUBSETS} UseSubset[s] = num_clusters; problem Master include 19. UseSubset Objective1 Partition Cardinality; use problem Master; 24. put Solving master problem, minimizing Objective1... ; 25. solve;

35 GRAPH PARTITIONING MILP SOLVER LOG NOTE: Problem generation will use 4 threads. NOTE: The problem has 2771 variables (0 free, 0 fixed). NOTE: The problem has 2771 binary and 0 integer variables. NOTE: The problem has 67 linear constraints (0 LE, 67 EQ, 0 GE, 0 range). NOTE: The problem has linear constraint coefficients. NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range). NOTE: The OPTMODEL presolver is disabled for linear problems. NOTE: The MILP presolver value AUTOMATIC is applied. NOTE: The MILP presolver removed 1092 variables and 0 constraints. NOTE: The MILP presolver removed 7089 constraint coefficients. NOTE: The MILP presolver modified 0 constraint coefficients. NOTE: The presolved problem has 1679 variables, 67 constraints, and constraint coefficients. NOTE: The MILP solver is called. NOTE: The parallel Branch and Cut algorithm is used. NOTE: The Branch and Cut algorithm is using up to 4 threads. Node Active Sols BestInteger BestBound Gap Time NOTE: The MILP solver added 40 cuts with 2752 cut coefficients at the root % 3 NOTE: Optimal. NOTE: Objective = 124.

36 GRAPH PARTITIONING SOLUTION FOR 14 CLUSTERS, BOUNDS ± 100, OBJECTIVE1

37 GRAPH PARTITIONING PROC OPTMODEL CODE: MILP SOLVER 1. /* L_infinity norm */ 2. var MaxDeviation >= 0; 3. min Objective2 = MaxDeviation; 4. con MaxDeviationDef {s in SUBSETS}: 5. MaxDeviation >= subset_deviation[s] * UseSubset[s]; put Solving master problem, minimizing Objective2... ; 8. MaxDeviation = max {s in SUBSETS} (subset_deviation[s] * UseSubset[s].sol); 9. solve with MILP / primalin;

38 GRAPH PARTITIONING MILP SOLVER LOG NOTE: Problem generation will use 4 threads. NOTE: The problem has 2772 variables (0 free, 0 fixed). NOTE: The problem has 2771 binary and 0 integer variables. NOTE: The problem has 2838 linear constraints (0 LE, 67 EQ, 2771 GE, 0 range). NOTE: The problem has linear constraint coefficients. NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range). NOTE: The MILP presolver value AUTOMATIC is applied. NOTE: The MILP presolver removed 1092 variables and 2733 constraints. NOTE: The MILP presolver removed constraint coefficients. NOTE: The MILP presolver added 1798 constraint coefficients. NOTE: The MILP presolver modified 329 constraint coefficients. NOTE: The presolved problem has 1680 variables, 105 constraints, and constraint coefficients. NOTE: The MILP solver is called. NOTE: The parallel Branch and Cut algorithm is used. NOTE: The Branch and Cut algorithm is using up to 4 threads. Node Active Sols BestInteger BestBound Gap Time % 2 NOTE: The MILP presolver is applied again % % 3 NOTE: Optimal. NOTE: Objective = 29.

39 GRAPH PARTITIONING SOLUTION FOR 14 CLUSTERS, BOUNDS ± 100, OBJECTIVE2

40 GRAPH PARTITIONING PAPER SAS Global Forum 2015 paper: Using the OPTMODEL Procedure in SAS/OR to Solve Complex Problems

41 PHARMACEUTICAL LAUNCH REVENUE OPTIMIZATION SCOTT SHULER

42 PRODUCT FORMULATION OPTIMIZATION IVAN OLIVEIRA

43 PRICING OPTIMIZATION SCOTT SHULER

44 TECHNICAL OVERVIEW AND FUTURE FEATURES PATRICK KOCH

45 OPTIMIZATION IN MACHINE LEARNING OVERVIEW 1. Training Predictive Models Optimization problem Parameters opt and train 2. Tuning Predictive Models Manual, grid search, optimization Optimization challenges 3. Distributed / parallel processing 4. Sample results

46 OPTIMIZATION IN MACHINE LEARNING PREDICTIVE MODELS Transformation of relevant data to high quality descriptive and predictive models x 1 w ij Ex.: Neural Network How to determine weights, activation functions? x 2 x 3 w jk f 1 (x) f 2 (x) How to determine model configuration? x 4... f m (x) x n Input layer Hidden layer Output layer

47 OPTIMIZATION IN MACHINE LEARNING MODEL TRAINING OPTIMIZATION PROBLEM Objective Function f w = 1 n i=1 n L w; x i, y i + λ 1 w 1 + λ 2 2 w 2 2 Loss, L(w; x i, y i ), for observation (x i, y i ) and weights, w Stochastic Gradient Descent (SGD) Variation of Gradient Descent w k+1 = w k η f(w k ) Replaces f w k with mini-batch gradient f t (w) = 1 m i=1 m L(w; x i, y i ) Several tricks to improve performance

48 OPTIMIZATION IN MACHINE LEARNING MODEL TRAINING OPTIMIZATION PARAMETERS (SGD) Learning rate, η Too high, diverges Too low, slow performance Momentum, μ Too high, could pass solution Too low, no performance improvement Regularization parameters, λ 1 and λ 2 Too low, has little effect Too high, drives iterates to 0 Other parameters Adaptive decay rate, β Mini-batch size, m Communication Frequency, c Annealing Rate, r

49 OPTIMIZATION IN MACHINE LEARNING HYPERPARAMETERS Quality of trained model governed by so-called hyperparameters No clear defaults agreeable to a wide range of applications Optimization options (SGD) Neural Net training options Number of hidden layers Number of neurons in each hidden layer Random distribution for initial connection weights (normal, uniform, Cauchy) Error function (gamma, normal, Poisson, entropy) x 1 x 2 x 3 x 4. w ij. w jk. f 1 (x) f 2 (x) f m (x) x n Input layer Hidden layer Output layer

50 OPTIMIZATION IN MACHINE LEARNING MODEL TUNING Traditional Approach: manual tuning Even with expertise in machine learning algorithms and their parameters, best settings are directly dependent on the data used in training and scoring Hyperparameter Optimization: grid vs. random vs. optimization x 2 x 2 x 2 x 1 Standard Grid Search x 1 Random Search x 1 Random Latin Hypercube

51 OPTIMIZATION IN MACHINE LEARNING HYPERPARAMETER OPTIMIZATION CHALLENGES T(x) Objective blows up Flat-regions. Node failure Categorical / Integer Variables Noisy/Nondeterministic Tuning objective, T(x), is validation error score (to avoid increased overfitting effect) x

52 OPTIMIZATION IN MACHINE LEARNING PARALLEL HYBRID DERIVATIVE-FREE OPTIMIZATION STRATEGY Default hybrid search strategy: 1. Initial Search: Latin Hypercube Sampling (LHS) 2. Global search: Genetic Algorithm (GA) Supports integer, categorical variables Handles nonsmooth, discontinuous space 3. Local search: Generating Set Search (GSS) Similar to Pattern Search First-order convergence properties Developed for continuous variables All easy to parallelize

53 TIme (minutes) Time (hours) OPTIMIZATION IN MACHINE LEARNING DATA PARALLEL & MODEL PARALLEL Credit Data 49k Train / 21k Validate Handwritten Data Train 60k/Validate 10k Data Parallel Distributed Data, Sequential Tuning 10 8 Data Parallel & Model Parallel Distributed Data, Parallel Tuning Number of Nodes for Training Maximum Models Trained in Parallel

54 Error % Objective (Misclassification Error %) OPTIMIZATION IN MACHINE LEARNING SAMPLE HYPERPARAMETER OPTIMIZATION RESULTS Default train, 9.6 % error Iteration 1 Latin Hypercube best, 6.22% error Iteration 1: Latin Hypercube Sample Error Handwritten Data Train 60k/Validate 10k Evaluation 2 Very similar configuration can have very different error Best after tuning: 2.61% error Iteration

55 OPTIMIZATION IN MACHINE LEARNING MANY TUNING OPPORTUNITIES Other Machine Learning algorithms SVM - Support Vector Machines Decision Trees Random Forest Gradient Boosting Trees Other search methods, extending hybrid solver framework Bayesian Optimization Surrogate-based Optimization Better (generalized) Models = Better Predictions = Better Decisions

56 THANK YOU!