European Journal of Operational Research

Transcription

1 Euopean Jounal of Opeational Reseach 198 (9) Contents lists available at ScienceDiect Euopean Jounal of Opeational Reseach jounal homepage: Stochastics and Statistics Contolled sequential factoial design fo simulation facto sceening Hua Shen a,1, Hong Wan b, * a Wachovia Copoation, Chalotte, NC , USA b School of Industial Engineeing, Pudue Univesity, West Lafayette, IN , USA aticle info abstact Aticle histoy: Received 1 Apil 7 Accepted Septembe 8 Available online 1 Septembe 8 Keywods: Simulation Compute expeiments Design of expeiments Facto sceening Sequential factoial design Sceening expeiments ae pefomed to eliminate unimpotant factos efficiently so that the emaining impotant factos can be studied moe thooughly in late expeiments. This pape poposes contolled sequential factoial design (CSFD) fo discete-event simulation expeiments. It combines a sequential hypothesis testing pocedue with a taditional (factional) factoial design to contol the Type I eo and powe fo each facto unde heteogeneous vaiance conditions. We compae CSFD with othe sequential sceening methods with simila eo contol popeties. CSFD equies few assumptions and demonstates obust pefomance with diffeent system conditions. The method is appopiate fo systems with a modeate numbe of factos and lage vaiances. Ó 8 Elsevie B.V. All ights eseved. 1. Intoduction Stochastic simulation is one of the most widely used techniques fo opeations eseach and management science. Hee the stochastic simulation efes to the analysis of stochastic pocesses though the geneation of sample paths (ealizations) of the pocesses (Hendeson and Nelson, 6). Simulation expeiments ae typically faste, cheape, and moe flexible than physical expeiments. They ae especially useful fo pilot studies of complicated systems whee the physical expeiments ae too expensive o time-consuming. Despite the pevalence of simulation as a decision-suppot tool, thee is a huge gap between the complexity of the systems unde investigation and the expeimental designs used to study these methods. Simulation models may take months o yeas to develop and have liteally thousands of components (factos) fo which the potential effects on system pefomance ae unknown. Sceening expeiments in this case ae desied to quickly eliminate those unimpotant factos fo a moe compact model and faste, moe tanspaent analysis. Many sceening methodologies have been developed to identify impotant factos with an economical numbe of obsevations. The most common ones ae factional factoial, cental composite, and Plackett Buman designs (Myes and Montgomey, ). Howeve, these sceening designs wee developed fo physical expeiments. They typically involve fewe than 5 factos and do not * Coesponding autho. Tel.: ; fax: addesses: hua.shen@wachovia.com (H. Shen), hwan@pudue.edu (H. Wan). 1 Tel.: ; fax: take advantage of the sequential popety of simulation expeiments. Pocedues developed fo stochastic simulation expeiments include one-facto-at-a-time designs (Campolongo et al., ); methods based on fequency domain analysis (Moice and Badhan, 1995); edge designs (Elste and Neumaie, 1995); iteated factional factoial designs (Campolongo et al., ) and the Tocine sceening pocedue (Tocine and Malone, 1). Kleijnen et al. (5) gives a geneal eview of the design and analysis of simulation expeiments. Unfotunately, the analysis methods they have used typically assume equal vaiances acoss diffeent facto settings a ae occuence fo simulations of complex system. Moe impotantly, none of these designs fo stochastic simulations give the eo contol we desie. Contolled sequential bifucation (CSB) (Wan et al., 3, 6) method is poposed to stess these poblems. It combines a hypothesis testing pocedue with the sequential bifucation famewok poposed by Bettonvil and Kleijnen (1997). CSB is a seies of steps in which goups of factos ae tested. If a goup of factos is consideed unimpotant, then evey facto in the goup will be consideed unimpotant. If the goup is consideed impotant, then it is split (bifucated) fo futhe testing. Each facto will eventually be classified as impotant o unimpotant. Whethe the desied eo contol can be achieved depends on the statistical test employed at each step. Given an appopiate testing pocedue, CSB can contol the Type I eo of each facto and powe fo each bifucation step unde heteogeneous vaiance conditions. The sequential natue of the method makes it a good fit fo simulation expeiments. CSB-X incopoates a fold-ove design into the CSB famewok to eliminate the effects of second-ode tems and give unbiased sceening esults of main-effects (Wan et al., 8). Examples show that CSB (and CSB-X) ae highly efficient fo lage-scale poblems /$ - see font matte Ó 8 Elsevie B.V. All ights eseved. doi:1.116/j.ejo.8.9.5

2 51 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) when impotant factos ae of small pecentage and clusteed, since it can eliminate unimpotant factos in goups. On the othe hand, the sequential bifucation famewok detemines that CSB is a one-at-a-time design. Simulation eplications geneated in pevious design points may not be useful in the late sceening stages and new simulation eplications ae usually needed afte each bifucation step. This makes the method inefficient in many scenaios. In addition, the CSB method has to assume known diection of effects, which is not ealistic in many scenaios. Wan and Ankenman (6) popose a diffeent appoach to contolled sceening, namely two-stage contolled factional factoial sceening (TCFF). The use selects an appopiate (factional) factoial design and collects a small numbe of obsevations at each design point in the fist stage. Moe obsevations will be collected fo those design points with lage vaiances in the second stage and the weighted aveage of all obsevations geneated at each design point is consideed as a single pseudo-obsevation following a non-cental t-distibution. The weight is the numbe of obsevations at each design point. The design is then teated as an uneplicated (factional) factoial design. Because of the use of factoial designs, TCFF can test inteaction effects and does not equie the diections of the effects to be known. TCFF and CSB-X ae tested unde diffeent scenaios. The esults show that CSB-X equies fewe eplications when only 1% of factos ae impotant while the TCFF method has bette efficiency when thee ae no less than 5% of the factos impotant. TCFF is also moe obust in efficiency acoss diffeent scenaios. In this pape, we popose a diffeent appoach of sequential factoial design fo stochastic simulation expeiments. We call it contolled sequential factoial design (CSFD). CSFD combines sequential hypothesis testing pocedues with a full o factional factoial design to povide simultaneous Type I eo and powe contol fo sceening esults unde heteogeneous vaiance conditions. Unlike CSB methods, which usually need to geneate new obsevations at each bifucation step, CSFD can utilize all peviously geneated obsevations in late sceening stages. In most cases, afte the fist few effects ae classified, no moe simulation uns ae needed. Diffeent fom TCFF, the computational effot of CSFD is equally allocated to all design points. This seemingly inefficient appoach tuns out to be supeio in many situations; numeical esults show that CSFD is moe effective than TCFF and CSB methods in many cases. The pape is oganized as follows: The undelying esponse model and the objective of sceening ae discussed in Section. Section 3 descibes CSFD method and its pefomance in detail. Section 4 pesents empiical evaluations of CSFD compaed to CSB-X and TCFF. In Section 5, CSFD is implemented in a semiconducto manufactuing system. Conclusions and futue eseach ae pesented in Section 6.. Model desciption Suppose thee ae in total of L factos. A geneal metamodel including all main-effects and inteactions is given as follows: y ¼ b þ XL b i z i þ XL i;j¼1;i<j b i;j z i z j þþb 1;...;L z 1 z...z L þ e: Hee b t ¼fb 1 ; b ;...; b 1;;...;L g ae the effect coefficients (t epesents tanspose ). The level settings, z ¼ðz 1 ; z ;...; z L Þ, ae assumed to be deteministic. The eo tem, e, on the othe hand, is a andom vaiable; in this pape we assume it is Noð; ðzþþ, whee ðzþ is unknown. In pactice, the model may include any subset of the effects. Usually if an inteaction exists, the main effects and lowe-ode inteactions of all factos involved should also be included. The objective of ou sceening pocedue is to classify inteested effects into two goups, impotant ones and unimpotant ones. Fo effects with coefficients 6 D, CSFD should have 6 a pobability to declae them as impotant; fo effects with coefficients P D 1, the powe of identifying them as impotant should be P c. Those effects whose coefficients fall between D and D 1 ae consideed impotant and we want CSFD to have easonable, though not guaanteed, powe to identify them. The paametes D and D 1 ae the thesholds of impotance and citicality, espectively, with D 1 > D. Specifically, D is the minimum change in the expected esponse that is pactically impotant, and D 1 is a change in the expected esponse that we would not want to miss. The a and c ae use-specified paametes. In pactice, when we conside whethe a change in the esponse is woth pusuing, the cost to poduce the change is meaningful. Wan et al. (3, 6) poposed a cost model to detemine the thesholds and facto settings so that all effects can be compaed with thesholds without ambiguity. The selections of levels and thesholds, on the othe hand, will not influence the pefomance of CSFD. Afte detemining the levels of factos, CSFD will then code them fom 1 to+1(montgomey, 5). 3. Contolled sequential factoial design (CSFD) The fist step of CSFD is to select a full o factional factoial design that will be sequentially implemented. Factoial designs ae one of the most widely used expeiment designs in pactice (see e.g., Montgomey, 5). Full factoial designs contain all combinations of facto settings and can independently estimate all main effects and inteactions. Factional factoial designs contain a subset of all combinations of facto settings and some effects estimates will be confounded. If only main-effects and low-ode inteactions ae of inteests, factional factoial designs ae sufficient. In this eseach, we focus on L full factoial designs and L p factional factoial designs, i.e., each facto has two levels. A factoial design is chaacteized by its design matix X. Each ow of the design matix specifies an expeiment setting of each facto, and each column specifies the settings of the specific effect acoss all expeiments. The esolution of a factional factoial design chaacteizes the degee of confounding. Fo example, a Resolution III design can independently estimate all main-effects, but some main effects ae confounded with two-facto inteactions. A Resolution V design has all main-effects and two-facto inteaction estimatos independent of each othe. The highe the esolution, the moe design points ae equied. The factoial design is analyzed though egession technique. Fo a linea egession model Y ¼ Xb þ e, whee X is the design matix, Y ¼fy 1 ; y ;...; y I g t is the obsevation vecto, e is the eo vecto, and the least squae estimato of the effect coefficients is ^b ¼ðX t XÞ 1 X t Y. Fo a full factoial design, all main and inteaction effects can be estimated independently; fo a factional factoial design, some of the effects ae confounded with othes. Many books (e.g. Wu and Hamada, ) and softwae packages povide ecommended designs of diffeent esolutions fo 3 5 factos. Fo lage-scale cases, Resolution III factional factoial designs can be easily constucted. Fo a L-facto main-effects model, we will simply use a m-facto full factoial design with m design points and confound the m þ 1toL factos main-effects with the inteactions of the fist m factos. Hee, m is an intege satisfying m 1 6 L < m (Montgomey, 5). Wan and Ankenman (6) descibed the methods to constuct Resolution IV designs. Fo constucting lage-scale Resolution V designs, please see Sanchez and Sanchez (5) CSFD pocedue The geneic stuctue of CSFD methods is pesented in Table 1. Random obsevations ae geneated in batches and each batch is

3 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) Table 1 Algoithm of CSFD Initialization: Ceate an empty goup fo effects of inteests. Add the effects fb 1 ; b ;...; b K g to the goup. Select a factoial design with design matix X. Geneate n eplications of obsevations ^bðjþ ¼ðX t XÞ 1 X t YðjÞ, j ¼ 1; ;...; n ; n 1 ¼ n ¼...¼ n K ¼ n ; N ¼ n While the goup is not empty, do Remove: emove an effect b k fom the goup. n k ¼ N. Qualified testing pocedue: calculate necessay statistics of ^b k. Collect n new eplications until specific eo contol equiements achieved fo b k (Table ). Update: Calculate ^bðn þ jþ ¼ðX t XÞ 1 X t YðN þ jþ; j ¼ 1; ;...; n; N ¼ N þ n. Classify all effects in the goup that can achieve the specific eo contol with N þ n: End While called one eplication. Each eplication contains M obsevations, one fom each design point (M is the numbe of design points), and can povide one estimate fo evey desied effect, i.e., fo the jth eplication, ^bðjþ ¼ðX t XÞ 1 X t YðjÞ. The initial numbe of eplications, n, is an use-specified intege numbe. Effects ae classified one at a time. Fo each effect, CSFD fist computes the obseved mean and vaiance of the estimated effect coefficient based on all cuent available eplications and detemines whethe thee ae enough eplications to daw a conclusion with the specified eo contol (using the qualified hypothesis testing pocedue discussed in Section 3.). If no conclusion can be made, moe eplications will be geneated until the sample size is lage enough to classify the cuent effect. It should be noticed that the sceening of the cuent effect will use all the peviously geneated eplications as initial sample. Theefoe, afte the fist effect, the initial numbe of eplications is usually geate than n. The notations fo CSFD ae given below. Thee ae in total L indexed factos and thee ae K effects of inteest, 1 6 K < L. b k : kth effect coefficient, k ¼ 1; ;...; K. The selected design has M design points. ^b i ðjþ: The estimate of the ith effect fom the jth obsevation, j ¼ 1; ;...; N; i ¼ 1; ;...; K. ^bðjþ ¼ð^b 1 ðjþ; ^b ðjþ;...; ^b K ðjþþ t. n : Numbe of initial eplications geneated at the beginning of the sceening pocedue. y m ðjþ: The jth obsevation of the mth design point, j ¼ 1; ;...; N; m ¼ 1; ;...; M. n k : Numbe of available eplications at the beginning of the classification of the kth effect. n 1 ¼ n. N: The cuent numbe of eplications. Fo simulation expeiments, it s usually unealistic to assume equal vaiances acoss diffeent design points, i.e., Vaðy m1 Þ Vaðy m Þ; m 1 ; m ¼ 1; ;...; M; m 1 m. Also, the obsevations acoss diffeent design points may be coelated with each othe (see the discussion of implementation of common andom numbes in Section 4.3). Howeve, as long as y m ðjþ; j ¼ 1; ;...n k ae i:i:d andom vaiables fo any given m, 1 6 m 6 M, the ^b k ðjþ; j ¼ 1; ;...; N will be i:i:d: andom vaiables fo any k. It should be noticed that the output data of simulation expeiments ae usually dependent andom vaiables. In these cases, we can use the batch mean method by gouping seveal obsevations into one batch and use the batch mean as the esponse. With appopiate batch size, these batch means can be appoximated as i:i:d: nomal andom vaiables (Alexopoulos, 6; Law and Kelton, ). Fom this pespective, the CSFD method is in fact an analysis method fo eplicated factoial designs with heteogeneous vaiances. The qualified hypothesis pocedue detemines the numbe of the eplications equied to achieve desied eo contol (in othe wods, the stopping ules of the sequential factoial design). Moe detail is discussed in the following section. 3.. Pefomance of CSFD The pocess to classify the impotance of desied effects is actually to sequentially test the following hypotheses: H : jb k j 6 D vs: H 1 : jb k j > D : The selection of the testing pocedue has a significant impact on the effectiveness and efficiency of CSFD. Wan et al. (6) intoduced the concept of a Qualified test. A testing pocedue is called qualified if it guaantees that P {Declae an effect impotant keffectj 6 D } 6 a, and P {Declae an effect impotant keffectj P D 1 } P c. Using this concept, Poposition 1 is staightfowad: Poposition 1. Given a qualified testing pocedue, CSFD guaantees that PfDeclae effect k impotantkb k j 6 D g 6 a; and PfDeclae effect k impotantkb k j P D 1 g P c; fo any k ¼ 1; ;...; K. In summay, given a qualified testing pocedue, CSFD povides simultaneous Type I eo and powe contol fo each effect while CSB only povides step-wise powe contol. Fom this pespective, CSFD is supeio. We use a specific qualified fully sequential test poposed by Wan et al. (6) and Wan et al. (8). The pocedue takes a small numbe of obsevations at the initial stage, and then adds one eplication at a time and teminates as soon as a conclusion can be made. Numeical esults show that the test is highly efficient in many cases. The stuctue of the fully sequential testing pocedue is given in Table along with the notations below: B k ¼ P N j¼1^b k ðjþ=n. S k ¼ P n k ð^b j¼1 k ðjþ B k Þ =ðn k 1Þ: Sample vaiance S k is computed based on the fist n k eplications. S k will not be updated when moe eplications ae geneated. k ¼ðD 1 D Þ=4. aðkþ ¼a S k. MðkÞ ¼baðkÞ=kc. Thee is no close-fom solution fo a and in geneal cases and the values of these two constants have to be obtained numeically Table Fully sequential testing pocedue (1) Set n ¼. () If N > MðkÞ, (a) if jb k j <, then classify the effect as unimpotant. (b) else classify the effect as impotant. (3) Else (i.e., N þ n 6 MðkÞ) (a) if ðn þ nþðjb k j Þ 6 kðn þ nþ aðkþ, then classify the effect as unimpotant. (b) else if ðn þ nþðjb k j Þ P aðkþ kðn þ nþ, then classify the effect as impotant. (c) else geneate one moe eplication; update jb k j; n ¼ n þ 1; go to ().

4 514 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) (see Appendix). Fo a special case whee a ¼ 1 c, (i.e., taget Type I eo equals taget Type II eo), a and ae given as following: g ¼ðexpðuÞ 1Þ=, whee u ¼ lnðaþ=ðn 1Þ. a ¼ gðn 1Þ=ðD 1 D Þ. ¼ðD þ D 1 Þ=. Again, since we assume that the esponses at each design point (fo example, the batch means) follow an i:i:d: nomal distibution (i.e., the esponse vectos of the n eplications ae i:i:d multivaiate nomal andom vaiables), ^b k ðjþ fo any j; k, which is a linea combinations of the esponses of all design points, is also nomally distibuted; and ^b k ð1þ; ^b k ðþ;...; ^b k ðn k Þ ae i:i:d: nomal andom vaiables fo any k. Theefoe, all the poofs of the qualification of the fully sequential testing pocedues in Wan et al. (6) and Wan et al. (8) ae also valid fo CSFD Illustated example To illustate the pocedue, lets look at a toy poblem with only two factos x 1 and x. The inteested effect coefficients ae b 1 ; b, and b 1;, and thei tue values ae, 3 and.5, espectively (fo simplicity, we set b ¼ ). We futhe set D ¼ and D 1 ¼ 3. A full factoial design with 4 uns ae selected with the design matix in Table 3. We assume the standad deviation at each design point equals 1 þ :5jesponsej. Moe specifically, the esponse at design point m equals y m ¼ b 1 x 1m þ b x m þ b 1; x 1m x m þ m ; whee m follows a nomal distibution with mean and standad deviation 1 þ :5jb 1 x 1m þ b x m þ b 1; x 1m x m j, with m ¼ 1; ; 3; 4. We can easily see that ^b1 ðjþ ¼½ y 1j þ y j y 3j þ y 4j Š=4, ^b ðjþ ¼ ½y 1j y j y 3j þ y 4j Š=4, and ^b 1; ðjþ ¼½ y 1j y j þ y 3j þ y 4j Š=4; j ¼ 1; ;...; N. We pick n ¼ 3, and simulated the esponses. The initial Y ¼½y ij Š ae listed in Table 4 and the accodingly estimated effects ae listed in Table 5. Table 3 Full factoial design Run b 1 b b 1; Table 4 Simulated esponses Design points Rep 1 Rep Rep Table 5 Estimated effects of ^b 1, ^b and ^b 1; ^b Rep 1 Rep Rep 3 Std ^b ^b ^b 1; ð1þ Table 6 aðkþ fo ^b 1, ^b and ^b 1; ^b ^b ^b 5.8 ^b 1; 9.49 We can see that the estimations can be significantly diffeent fom the tue values. Given a ¼ 1 c ¼ :5, we can calculate the citical values aðkþ of the fully sequential test fo b 1, b and b 1; (Table 6). Fo all thee effects, ¼ :5 and k ¼ :33. The fully Lowe Bound Uppe Bound aðkþ Lowe Bound Uppe Bound Lowe Bound Uppe Bound Fig. 1. Sceening of b 1 ; b and b 1;.

5 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) sequential test monitos whethe the patial summation P j¼n ð ^b k ðjþ Þ difts out of a tiangula egion with uppe bound = aðkþ k and lowe bound = aðkþþk, whee is the sample size. Fo this specific example, we can veify that no conclusion can be made fo any of the thee effects with the initial thee eplications. Theefoe the expeiments continued. Each eplication contains thee obsevations, one fom each design point. It tuns out that b 1 is classified as unimpotant with six eplications (thee afte the initial n obsevations); b is classified as impotant with 8 eplications, and b 1; is classified as unimpotant with fou eplications. The boundaies and patial summations ae demonstated in Fig. 1. This example also shows that the sequence of inspecting effects ae ielevant. Fo example, if b is the fist one to be classified, then no eplications will be collected fo classifying b 1 and b 1;. 4. Empiical evaluation In this section, we pesent the numeical esults of atificial examples to compae CSFD with TCFF and CSB (CSB-X) methods. Because the pefomance of CSB-X is usually as good as, if not bette than, CSB (Wan et al., 8), we will only un CSB-X in the compaison. The fully sequential test is used fo both CSB-X and CSFD. In all cases, nomal eos ae assumed with mean and standad deviation ¼ a ð1 þje½yšjþ, i.e., the standad deviation is popotional to the expected esponse. The vaiance paamete a takes diffeent values at diffeent cases. Common andom numbes wee not employed (the effect of CRN is discussed in Section 4.3). Fo each case consideed, all thee methods ae epeatedly applied 1 times and the pecentage of times each effect is classified as impotant is ecoded, which is denoted by PðDIÞ; fo the kth effect, this pecentage is an unbiased estimate of P{Declae effect k impotant}. We use # of uns to epesent the aveage numbe of simulation uns equied fo each method. Table 7 Paametes fo small-scale case with inteactions Paamete Value L 1 D D 1 4 a.5 c.95 a ð1þje½yšjþ a.1,.3, Small-scale case with two-facto inteactions We fist compae the effectiveness and efficiency of CSFD, CSB- X and TCFF methods on a small-scale case (facto numbe L ¼ 1) with second-ode inteactions. The simulation models in this scale (o smalle) can be found in vaious disciplines. Fo example, a hospital scheduling simulation model with fou factos ae discussed in Kopach et al. (7). The simulation paametes ae listed in Table 7 and the effect coefficients of the model ae set up as follows: Main-effects: ðb 1 ; b ;...; b 1 Þ¼ð:; :; :4; :7; 3:; 3:3; 3:6; 3:8; 4:; 4:Þ. Inteaction effects: ðb 1; ; b 3;5 ; b 4;8 Þ¼ð:; 4:; 4:Þ. All othe second o highe ode inteactions ae zeo. A 1 3 V design is used in CSFD and TCFF. Fo CSB-X, facto effects ae tested in goups and the design points ae detemined sequentially (Wan et al., 8). All thee methods ae epeatedly un with diffeent n (Recall that fo CSFD and TCFF, n is the initial numbe of eplications at each design point; fo a 1 3 V factoial design each eplication contains one obsevation fom 1 3 ¼ 18 design points; fo CSB-X, n is the initial numbe of eplications geneated at each design point). The selection of n has little influence on the sceening esults but does change the total simulation uns equied. Table 8 pesents the esult of each case with smallest aveage numbe of simulation uns equied. CSFD and TCFF have simila efficiency, although CSFD is a little moe efficient when the vaiance is small. We can also see when the vaiance is small, CSB-X can be moe efficient than CSFD and TCFF; howeve, when the vaiance gets lage, the simulation effot equied by CSB-X inceases damatically and CSFD and TCFF ae much moe efficient than CSB-X. Fo all thee cases, PðDIÞs should be less than a ¼ :5 fo effects b 1 ; b ; b 3, and b 1; since thei coefficients ae 6 D, and geate than c ¼ :95 fo effects b 9 ; b 1 ; b 3;5, and b 4;8 since thei coefficients ae P D 1. We can see that CSFD stictly meets both equiements and is consevative. CSB-X has consevative Type I eos but the powes ae lowe fo citical factos when vaiances ae lage. TCFFs esults ae close to the specified eo contol. CSFD and TCFF show no diffeence in classifying main-effects and inteaction effects. 4.. Lage-scale cases As discussed in Saege and Hinch (1), the Amy uns a seies of simulation expeiments to geneate tactics and simultaneously Table 8 Small-scale case with nd-ode inteactions Effect a ¼ :1 a ¼ :3 a ¼ 1: CSFD TCFF CSB-X CSFD TCFF CSB-X CSFD TCFF CSB-X b 1 ¼ : b ¼ : b 3 ¼ : b 4 ¼ : b 5 ¼ 3: b 6 ¼ 3: b 7 ¼ 3: b 8 ¼ 3: b 9 ¼ 4: b 1 ¼ 4: b 1; ¼ :..55 n/a..48 n/a.3.5 n/a b 3;5 ¼ 4: n/a n/a n/a b 4;8 ¼ 4: n/a n/a n/a n # of uns

6 516 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) assess the value of vaious systems. The simulations that ae used ae vey lage in size, sometimes taking many hous to un a single expeiment and having a lage numbe of use selectable input values. We will conside two lage-scale cases with factos and 5 factos, espectively. Main-effects models ae assumed in both cases. The simulation paametes of the lage-scale cases ae given in Table 9. We use a Resolution III factional factoial designs discussed in Section 3. The size of the design is 56 fo -facto case and 51 fo 5-facto case. In both cases, the effect coefficients ae set up as follows: 5% of the factos ae assumed to be citically impotant and thei effect coefficients equal to D 1 ¼ 4. Thee ae anothe 5% of factos whose effect coefficients ae set to be D ¼. All othe factos have effect coefficients set to be. Fo each case, thee scenaios of facto distibution will be studied: The fist scenaio has all factos with non-zeo effect coefficients clusteed togethe with the smallest indices so that the numbe of impotant goups is as small as possible at each step of CSB-X. The second scenaio has factos with non-zeo effect coefficients evenly spead so thee ae maximum numbe of impotant goups emaining at each step. The thid scenaio has those factos andomly spead. Again, vaious initial numbes n ae used. Table 1 pesents the esults of the most efficient case fo each scenaio. We can daw simila conclusions on these thee methods as in small-scale case and the efficiency advantage of CSFD is moe obvious. When vaiance inceases, CSB-X becomes vey inefficient. Table 9 Paametes fo lage-scale cases Paamete Value L, 5 D D 1 4 a.5 c.95 a ð1þje½yšj) a.1,.1,.3, Summay and futhe discussions Efficiency: The numeical evaluation shows that oveall CSFD has supeio pefomance compaed to TCFF o CSB-X. Compaed with CSB-X, CSFD is moe efficient in most cicumstances (except fo cases with vey small vaiances). CSFD has simila efficiency as TCFF in small-scale case, but is moe effective than TCFF in lage-scale cases. This seems counte-intuitive since in lage-scale cases, each eplication of CSFD equies a lage numbe of simulation uns. Howeve, CSFD usually only needs a few eplications to sceen all effects. This is because the vaiances of the effect estimates ae small. Conside CSFD with a L full factoial design and assume equal vaiances ð Þ acoss the esponse suface. It is easy to veify that Vað^b k Þ¼ = L, k ¼ 1; ;...; K. Fo CSB-X, the vaiance of a goup effect estimate will be = > = L in the same situation. Thus, compaed to CSB-X, CSFD significantly educes the vaiances of the effect estimates in most cases. This is why CSFD is not as sensitive to the change of vaiance as CSB-X and is oveall moe effective. Compaed with TCFF, which may equie a lage numbe of simulation uns at design points with lage vaiances, CSFD takes equal numbes of uns at each design point and the esulting effect estimates shae the same vaiance. Initial sample size selection: Fig. demonstates the elationship between the initial eplication numbe n and the aveage simulation uns equied fo sceening (CSFD, -facto case, 3d scenaio, and a ¼ 1:). This figue is consistent with Fig in Kim and Nelson (6), which pesents the typical fom of expected simulation uns equied as a function of initial numbe n in anking-and-selection poblems. Thee often exists an unknown optimal initial sample size; an n too small may lead to huge penalty (as shown in Fig. ) and an n too lage is usually unnecessay. The eason is that when n is too small, it may poduce a lage sample vaiance, which futhe leads to a lage total numbe of eplications. When n is too lage, all of the sequential testing pocedues each conclusions in the fist stage with moe than necessay obsevations. This is why the ight side of the cuve is actually a staight line. As the numeical esults show, with othe system paametes fixed, the lage the vaiance, usually the lage the optimal n is. A small deviation fom the tue optimal initial numbe usually has little influence on the simulation effot. Common andom numbes: The impact of common andom numbes (CRN) in CSFD is discussed as follows. Fo a L full factoial design, ^b k ¼ 1 X t L C k Y and the enties of X t C k ae equal numbe of 1s and 1s. Suppose X t C k ¼ða 1 ; a ;...; a LÞ, whee a i ¼þ1o 1, then Table 1 Simulation uns in lage-scale cases Case Scenaio CSFD TCFF CSB-X CSFD TCFF CSB-X a =.1 a =.1 -facto 1st nd d n /14/14 5-facto 1st nd d n 4 4 5/4/ /3/35 a =.3 a = 1. -facto 1st >1, nd >1, 3d >1, n /35/ facto 1st >1,, nd >1,, 3d >1,, n /5/6 4 Total Numbe of Simulation Runs Initial Numbe of Replication n Fig.. Aveage simulation uns vs. initial numbe n.

7 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) Vað^b k Þ¼ 1 L X Va L ¼ 1 ¼ 1 L X L L X L a i Y i! a i VaðY iþþ X i j VaðY i Þþ X i j a i a j CovðY i ; Y j Þ a i a j CovðY i ; Y j Þ The effect of the CRN is detemined by the magnitude of each CovðY i ; Y j Þ. In a special case whee all covaiances ae equal to m >, since thee ae L 1 positive a i s and L 1 negatives a i s in X t C k, L ð L 1 Þ a i a j s will be negative and the othe L ð L 1 1Þ a i a j s will be positive. Theefoe Vað^b k Þ¼ 1 L X L ¼ 1 X L L VaðY i Þ L CovðY 1 ; Y Þ VaðY i Þ 1 L m; and 1 L m is the benefit of implementing CRN. Because of the stuctue of contasts, when the covaiances ae not too much diffeent fom each othe, the effect of CRN will be favoable. 5. Case study Wan et al. (6) and Wan et al. (8) have implemented CSB and CSB-X, espectively, in a simplified semiconducto manufactuing system to identify impotant machines and tanspotes that ae wothy of investment. In this section, we will implement!! :! CSFD with fully sequential testing pocedue on the system. Hee we will only give a bief desciption of the system. Fo details, please efe to Wan et al. (6) and Wan et al. (8). As shown in Fig. 3, the semiconducto manufactuing system consists of two basic steps, diffusion and lithogaphy, each of which contains multiple stations. Raw mateial is eleased into the system at the ate of 1 cassette/hou and pocessed in singlecassette loads. It begins with the diffusion pocess and then poceeds to the lithogaphy pocess. Fo each poduct, the two pocesses altenate until all passes equied ae completed. The pecentages of the poduct mix and the numbe of passes of each poduct type equies ae given in Table 11. Tanspotes (AGV and CONVEYOR) move poducts between diffusion and lithogaphy and ae also consideed as stations. The time of moving poducts within each pocess step is negligible. The mean pocessing times of each cassette at diffeent stations ae listed in Table 1. The pefomance measue of the system is the mean cycle time of poduct pocessing weighted by the pecentages of diffeent poducts. The factos of inteests ae the numbes of fast and slow machines Table 11 Poduction mix and passes Poduct types Pecentage (%) Passes equied A 15 B C 3 1 D 1 Table 1 Mean pocessing time pe cassette at each station (hous) Station Fast machine Slow machine CLEAN LOAD QUARTZ OXIDIZE UNLOAD QUARTZ TEST COAT STEPPER DEVELOP TEST AGV.8 n/a CONVEYOR n/a.19 Table 13 Facto desciption and settings (unit numbe) Facto id Facto desciption Low High Fig. 3. Poduction pocess of the semiconducto manufactuing system (Wan et al., 6, 8). 1 Numbe of slow machines in OXIDIZE 9 94 Numbe of fast machines in STEPPER 3 Numbe of fast machines in COAT 4 4 Numbe of slow machines in CLEAN Numbe of fast machines in TEST Numbe of fast machines in TEST 4 7 Numbe of slow machines in STEPPER Numbe of slow machines in COAT Numbe of fast machines in CLEAN 4 1 Numbe of slow machines in TEST Numbe of slow machines in TEST Numbe of slow machines in LOAD QUARTZ Numbe of slow machines in UNLOAD QUARTZ Numbe of fast machines in LOAD QUARTZ 1 15 Numbe of fast machines in UNLOAD QUARTZ 1 16 Numbe of AGVs 1 17 Numbe of slow machines in DEVELOP Numbe of CONVEYORS Numbe of fast machines in OXIDIZE Numbe of fast machines in DEVELOP 6

8 518 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) Table 14 Sceening esults of CSFD (a ¼ :5; c ¼ :95) ðd ; D 1 Þ at each station and the numbes of each kind of tanspote. The simulation of the semiconducto manufactuing system was done in simlib (Law and Kelton, ). Fo each obsevation of mean cycle time, 365 days of opeations wee simulated with a 3-hou wam-up peiod. Table 13 gives the facto desciption and the level settings, which we pick the exteme levels as in Wan et al. (8). A factoial design is used, which equies 51 expeimental points in each eplication, such that no main and second-ode inteaction effects ae confounded with each othe. The initial sample size is set to be 3. The esult fo diffeent combinations of ðd ; D 1 Þ ae given in Table 14. The two-facto inteactions ae epesented by two factos in paenthesis. Fo example (13, 15) epesents the inteactions between facto 13 and 15. The sceening esults ae oveall consistent with the CSB and CSB-X sceening esults epoted in Wan et al. (6) and Wan et al. (8). Main effect of facto 16 is identified as the most impotant effect. When the thesholds ae educed, additional effects ae identified as impotant. When the thesholds ae educed to (.5, 1.), CSFD gives insightful infomation on second-ode inteactions, which is not available in othe methods. All fou cases equie 1536 eplications, which implies that the initial eplications have been sufficient to daw conclusions in all cases. 11 V 6. Conclusions and futue eseach CSFD is a sequential facto sceening appoach that povides simultaneous Type I eo and powe contol. It equies few assumptions on the model and little pio infomation about the system. With the option of using factional factoial designs, CSFD can handle lage-scale cases efficiently. The pefomance is obust acoss diffeent facto configuations. Numeical evaluation shows that thee is a complementay elationship between CSFD and CSB- X. That is, the weakness of CSB-X is usually the stength of CSFD, and vice vesa. Ou futue eseach will concentate on developing a hybid method combining CSB-X and CSFD methods togethe to take the advantages of both methods. In addition, the eplications collected at each design point will allow us to estimate not only the means, but also the vaiances. In the futue we ae inteested in exploing sceening designs fo obustness studies. Acknowledgements This eseach was patially suppoted by Pudue Reseach Foundation (PRF) and Naval Postgaduate School Gant No C-. Peliminay esults fom this pape appeaed in Shen and Wan (5) and Shen (6). Additional thanks go to Pofesso Bay L. Nelson and Pofesso Buce E. Ankenman fom Nothwesten Univesity fo thei insightful comments. Appendix Impotant effects identified (, 4) {16} (1.5, 3) {5,6,16,} (1, ) {3,5,6,14,15,16,} (.5, 1) {,3,5,6,9,1, 13, 14,15,16,17,18,, (1,14), (13,15), (16,18), (17,)} Fo the fully sequential testing pocedue intoduced in 3., when a 1 c, constants a and ae the solutions of the following equations: Z 1 Z 1 and 1 Z 1 Z 1 1 cðyþ hðxþ 1 cðyþ / yhðxþ ð D Þ f ðx; n 1Þdxdy ¼ a; hðxþ cðyþ hðxþ 1 cðyþ / yhðxþ ðd 1 Þ f ðx; n 1Þdxdy hðxþ ¼ 1 c; ð3þ p whee cðyþ ¼e ky, /ðxþ ¼e x = = ffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffi p p, hðxþ ¼ n 1= xa =k, and ( 1 f ðx; nþ ¼ n Cð nþ xn 1 e x ; x P ; x < ; which is the pobability density function of v distibution with n degees of feedom, and Cð nþ¼r 1 xn= 1 e x dx Numeical methods ae needed to evaluate Eqs. () and (3) and seach fo solutions. The following lemma can be poved, which makes the seach elatively easy (Wan et al., 8). Lemma 1. Fo a fixed, both powe and Type I eo ae deceasing in. Lemma. If the equied a 6 1 and c P 1, then D 6 6 D 1. Refeences Alexopoulos, C., 6. A compehensive eview of methods fo simulation output analysis. In: Peone, L.F., Wieland, F.P., Liu, J., Lawson, B.G., Nicol, D.M., Fujimoto, R.M. (Eds.), Poceedings of the 6 Winte Simulation Confeence. Institute of Electical and Electonics Enginees, Piscataway, NJ, pp Bettonvil, B., Kleijnen, J.P.C., Seaching fo impotant factos in simulation models with many factos: Sequential bifucation. Euopean Jounal of Opeational Reseach 96 (1), Campolongo, F., Kleijnen, J.P.C., Andes, T.,. Sceening methods. In: Saltelli, A., Chan, K., Scott, E.M. (Eds.), Sensitivity Analysis. John Wiley & Sons, New Yok, pp Elste, C., Neumaie, A., Sceening by confeence designs. Biometika 8 (3), Hendeson, S.G., Nelson, B.L., 6. Stochastic simulation. In: Hendeson, S.G., Nelson, B.L. (Eds.), Elsevie Handbooks in Opeations Reseach and Management Science: Simulation. Elsevie, pp Kim, S.-H., Nelson, B.L., 6. Selecting the best system. In: Hendeson, S.G., Nelson, B.L. (Eds.), Elsevie Handbooks in Opeations Reseach and Management Science: Simulation. Elsevie, pp Kleijnen, J.P.C., Sanchez, S.M., Lucas, T.W., Cioppa, T.M., 5. A use s guild to the bave new wold of simulation expeiments. INFORMS Jounal on Computing 17 (3), Kopach, R., DeLauentis, P.-C., Lawley, M., Muthuaman, K., Ozsen, L., Radin, R., Wan, H., Willis, D., 7. Effects of clinical chaacteistics on successful open access scheduling. Health Cae Management Science 1 (), Law, A.M., Kelton, W.D.,. Simulation Modeling and Analysis, thid ed. McGaw- Hill, New Yok. Montgomey, D.C., 5. Design and Analysis of Expeiments, sixth ed. John Wiley & Sons, New Yok. Moice, D.J., Badhan, I.R., A weighted least squaes appoach to compute simulation facto sceening. Opeations Reseach 43 (5), Myes, R.H., Montgomey, D.C.,. Response Suface Methodology: Pocess and Poduct Optimization Using Designed Expeiments, second ed. John Wiley & Sons, New Yok. Saege, K., Hinch, J., 1. Undestanding instability in a complex deteministic combat simulation. Militay Opeations Reseach 6 (4), Sanchez, S.M., Sanchez, P.J., 5. Vey lage factional factoial and cental composite designs. ACM Tansactions on Modeling and Compute Simulation 15 (4), Shen, H., 6. Efficient Facto Sceening in Simulation Expeiments. Ph.D. Dissetation, School of Industial Engineeing, Pudue Univesity, West Lafayette, IN. Shen, H., Wan, H., 5. Contolled sequential factoial design fo simulation facto sceening. In: Kuhl, M.E., Steige, N.M., Amstong, F.B., Joines, J.A. (Eds.), Poceedings of the 5 Winte Simulation Confeence. Institute of Electical and Electonics Enginees, Piscataway, NJ, pp Tocine, L., Malone, L., 1. An oveview of newe, advanced sceening methods fo the initial phase in an expeimental design. In: Petes, B.A., Smith, J.S., Medeios, D.J., Rohe, M.W. (Eds.), Poceedings of the 1 Winte Simulation Confeence. Institute of Electical and Electonics Enginees, Piscataway, NJ, pp ðþ

9 H. Shen, H. Wan / Euopean Jounal of Opeational Reseach 198 (9) Wan, H., Ankenman, B.E., 6. Two-stage contolled factional factoial sceening fo simulation expeiments. Jounal of Quality Technology 39 (), Wan, H., Ankenman, B.E., Nelson, B.L., 3. Contolled sequential bifucation: A new facto-sceening method fo discete-event simulation. In: Chick, S., Sánchez, P.J., Dein, D., Moice, D.J. (Eds.), Poceedings of the 3 Winte Simulation Confeence. Institute of Electical and Electonics Enginees, Piscataway, NJ, pp Wan, H., Ankenman, B.E., Nelson, B.L., 6. Contolled sequential bifucation: A new facto-sceening method fo discete-event simulation. Opeations Reseach 54 (4), Wan, H., Ankenman, B.E., Nelson, B.L., 8. Simulation facto sceening with contolled sequential bifucation in the pesence of inteactions. INFORMS Jounal on Computing, in pess. Wu, J.C.F., Hamada, M.,. Expeiments: Planning, Analysis, and Paamete Design Optimization. John Wiley & Sons, New Yok.