A Comparison of Two Parallel Ranking and Selection Procedures

Transcription

1 A Comparison of Two Parallel Ranking and Selection Procedures Eric C. Ni Shane G. Henderson School of Operations Research and Information Engineering, Cornell University Susan R. Hunter School of Industrial Engineering, Purdue University Dragos Florin Ciocan INSEAD Winter Simulation Conference, Savannah, GA December 8, 2014

2 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 1/18

3 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 2/18

4 Ranking and Selection (R&S) max i S µ i = E[Y (i; ξ)] Optimize a function through a stochastic simulation. Feasible region is finite: k = S <. User-specified parameter δ. Types of statistical guarantee (assuming µ 1... µ k ): Correct selection (CS): P[select system k µ k µ k 1 + δ] 1 α; Good selection (GS): P[select system i : µ i µ k δ] 1 α. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 3/18

5 Paths to GS guarantee GS procedures: Multiple comparisons with the best (Rinott, 1978; Nelson and Matejcik, 1995; Nelson et al., 2001) Simultaneous confidence intervals on µi max j i µ j of width δ for all i. Best-arm selection algorithms (Jamieson et al., 2013; Jamieson and Nowak, 2014) Guarantees to find the true best system if it is unique. CS procedures: Sequential screening (Paulson, 1964; Fabian, 1974; Kim and Nelson, 2001, 2006; Hong, 2006). The assumption µ k µ k 1 + δ is essential in proving statistical validity. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 4/18

6 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18

7 Previous work on parallel R&S Web services-based parallel simulation optimization (Yoo et al., 2009; Luo et al., 2000) Parallel version of a screening-based procedure, with asymptotic CS guarantee (Luo et al., 2013) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18

8 A parallel R&S procedure NHH Ni et al. (2013) Stage 0: Simulate all systems to estimate simulation completion times. Stage 1: (If variances need to be estimated) Independently of Stage 0, systems are simulated in parallel to obtain variance estimates. Stage 2: Remaining systems are iteratively simulated and screened (both in parallel) until one system remains. Highly scalable. Inherits CS guarantee from its sequential predecessor (Hong, 2006). Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 6/18

9 (Parallel) Procedure NSGS Two-stage procedure proposed by Nelson et al. (2001). Stage 0: We simulate all systems to estimate simulation completion times. Stage 1: (If variances need to be estimated) Independently of Stage 0, systems are simulated in parallel to obtain variance estimates. Perform a round of screening (in parallel) with Stage 1 output. Stage 2: (The Rinott step) Simulate up to (hs i /δ) 2 replications of each system i and choose the system with the highest sample mean. Parallel implementation inherits GS guarantee from Nelson et al. (2001), as the Rinott step leads to MCB confidence intervals. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 7/18

10 Procedure complexity Let i := µ k µ i. The expected number of replications required to eliminate a system i by system k is approximately O(σi 2 2 i log( 2 i )) using best-arm algorithms (Jamieson et al. (2013)) O((σ 2 i + σ i σ k )[max( i, δ)] 1 δ 1 ) using NHH O(σ 2 i δ 2 ) using NSGS (and its parallel version) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 8/18

11 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18

12 Numerical example Our parallel procedures are applied to a throughput-maximization problem (SimOpt.org). Table 1: Summary of three instances of the test problem Number of Highest Num. of systems in [µ k δ, µ k ] Instance systems k mean µ k δ = 0.01 δ = 0.1 δ = Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18

13 Ni, Henderson, Hunter and Ciocan Numerical example 1.0 Instance 1: 3249 systems 3.0 Instance 2: systems Standard Deviation σ Standard Deviation σ Mean µ Mean µ Figure 1: Mean-standard deviation profiles of two instances Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 10/18

14 Procedure Performance Table 2: Summary of procedure costs on 3 instances of the throughput maximization problem with α = 0.05, n 0 = 20 Total simulation Total Per Number of replications running replication systems k n 1 δ Procedure ( 10 6 ) time (s) time (µs) NHH NSGS NHH NSGS NHH NSGS 1, , NHH NSGS NHH 3, , NSGS N/A (too costly) 0.1 NHH NSGS 2, , Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 11/18

15 NHH procedure NSGS procedure Wallclock time (s) Perfect scaling Actual performance Number of workers Wallclock time (s) Perfect scaling Actual performance Number of workers Figure 2: Scaling result on 57,624 systems, δ = 0.1 Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 12/18

16 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18

17 A parallel R&S procedure with PGS Two main stages: Stage 1: (In parallel) Simulate and periodically screen systems until one system remains, or a pre-specified termination criterion is met. The screening method and termination criterion are jointly chosen such that P[System k survives Stage 1] is sufficiently large, say 1 α/2. Stage 2: (The Rinott step) Simulate up to (hs i /δ) 2 replications of each system i and choose the system with the highest sample mean. Stage 2 guarantees good selection amongst those surviving Stage 1. Theorem 1 The new procedure provides a good selection guarantee P[select system i : µ i µ k δ] 1 α if simulation outcomes Y (i; ξ) Normal(µ i, σ 2 i ). Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18

18 Implementing the PGS procedure MapReduce Standard distributed programming model Map() procedure to process data in parallel Simulate surviving systems Reduce() procedure to summarize Screen using the additional statistics Easy to program but incurs higher overhead Runs on the cloud (e.g. Amazon EC2) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 14/18

19 Procedure Performance Table 3: Summary of procedure costs Total simulation Total Per Number of replications running replication systems k n 1 δ Procedure ( 10 6 ) time (s) time (µs) NHH NSGS GS NHH NSGS GS , NHH NSGS 1, , GS NHH NSGS GS ,878 9,400 NHH and NSGS on XSEDE high-performance cluster GS on Amazon EC2 using Hadoop MapReduce Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 15/18

20 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 16/18

21 Conclusion Efficiency of procedures is sensitive to problem configuration and δ. NHH tends to cost significantly fewer simulation replications then parallel NSGS in many instances, but NSGS guarantees good selection. A new procedure with provable good selection guarantee. Saves simulation effort in some cases. Extends to cloud computing platforms. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 16/18

22 Thank you! Questions? 17/18

23 Sketch of Proof: Theorem 1 Let B denote a Brownian motion with drift. Lemma (Hong (2006), Theorem 1) Let m(r) and n(r) be nondecreasing functions of r = 1, 2,... and i, i be any two systems. Define Z(m, n) = [ σ 2 i /m + σ 2 i /n] 1 [ Xi (m) X i (n)] and Z (m, n) = B µi µ i ([σ 2 i /m + σ 2 i /n] 1 ). Then the random sequences Z(m(r), n(r)) and Z (m(r), n(r)) have the same joint distribution. Lemma P[min 0 t T B 0 (t) < A] = 2P[B 0 (T) < A] = 2 Φ(A/ T) for all A, T > 0. Based on the above, we can design procedures to limit the probability of eliminating the best system in Stage 1. 18/18

24 References I V. Fabian. Note on anderson s sequential procedures with triangular boundary. Annals of Statistics, 2(1): , L. Jeff Hong. Fully sequential indifference-zone selection procedures with variance-dependent sampling. Naval Research Logistics (NRL), 53(5): , K. Jamieson and R. Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In Information Sciences and Systems (CISS), th Annual Conference on, pages 1 6, ID: 1. Kevin Jamieson, Matthew Malloy, Robert Nowak, and SÃľbastien Bubeck. lil ucb: An optimal exploration algorithm for multi-armed bandits. arxiv preprint arxiv: , References 1/4

25 References II Seong-Hee Kim and Barry L. Nelson. A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation, 11(3): , Seong-Hee Kim and Barry L. Nelson. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research, 54(3): , Jun Luo, Jeff L. Hong, Barry L. Nelson, and Yang Wu. Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Working Paper, References 2/4

26 References III Yuh-Chuyn Luo, Chun-Hung Chen, E. Yucesan, and Insup Lee. Distributed web-based simulation optimization. In Proceedings of the 2000 Winter Simulation Conference, volume 2, pages , B. L. Nelson and F. J. Matejcik. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science, 41(12): , Barry L. Nelson, Julie Swann, David Goldsman, and Wheyming Song. Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research, 49(6): , References 3/4

27 References IV Eric C. Ni, Susan R. Hunter, and Shane G. Henderson. Ranking and selection in a high performance computing environment. In Proceedings of the 2013 Winter Simulation Conference, pages , E. Paulson. A sequential procedure for selecting the population with the largest mean from k normal populations. Annals of Mathematical Statistics, 35(1): , Yosef Rinott. On two-stage selection procedures and related probability-inequalities. Communications in Statistics - Theory and Methods, 7(8): , Taejong Yoo, Hyunbo Cho, and Enver Yücesan. Web services-based parallel replicated discrete event simulation for large-scale simulation optimization. Simulation, 85(7): , July References 4/4