ACCURACY AND EFFICIENCY OF MONTE CARLO METHOD. Julius Goodman. Bechtel Power Corporation E. Imperial Hwy. Norwalk, CA 90650, U.S.A.

Transcription

1 ACCURACY AND EFFICIENCY OF MONTE CARLO METHOD Julius Goodman Bechtel Power Corporation E. Imperial Hwy. Norwalk, CA 90650, U.S.A. ABSTRACT The accuracy of Monte Carlo method of simulating distributions is analyzed. It is shown that claimed accuracy of some standard computer codes is overstated. The correct formula for evaluation of the accuracy of any percentile (or fractile) is developed. It has been shown that increasing the accuracy of the Monte Carlo method by increasing the number of simulations is not economical because the time complexity of algorithm is inversely proportional to the squared relative error. Therefore, the new method of increasing the accuracy with a significant reduction in the number of simulations was developed. For the new method, the number of simulations and time complexity of algorithm are inversely proportional to the relative error. 1. INTRODUCTION The accuracy of the Monte Carlo method of assessment simulating distributions in probabilistic risk assessment (PRA) is significantly lower than what is widely believed. Some computer codes for which the claimed accuracy is about 1 percent for several thousand simulations, actually have 20 to 30 percent accuracy. Using these codes to compare the reliability or unavailability of two systems can lead to miscalculation and misjudgment. The purpose of this paper is to give a better estimate of accuracy of the traditional Monte Carlo Method in PRA and to develop a more efficient method of Monte Carlo simulation. 2. COMPARISON OF CLAIMED AND REAL ACCURACY OF MONTE CARLO COMPUTER CODES SIMULATION DISTRIBUTIONS Several computer codes were tested on claimed accuracy of simulated distributions. The test program considered simulation of a random function comprised of one or the product of several lognormal distributions. The accuracy of codes was determined by comparing simulated and exact analytical values. The results of all codes were similar; therefore, we consider here only the SAMPLE code widely used in Reactor Safety Study [1]. The claimed accuracy of SAMPLE is shown in Table 1. According to Reactor Safety Study, there is no connection between the accuracy of SAMPLE and the error factor of simulated random function. To make our point clearer, we simulated a spread distribution with a median equal to 1 x 10 and an error factor equal to corresponding to the lognormal

2 Table 1 Claimed Accuracy of SAMPLE Sample Size Accuracy on 95% Confidence Interval 1.0% 0.7% 0 5% parameter a=3.0c. The comparison of simulated and exact analytical calculations is given in Table 2. For the sample size 1200, claimed accuracy 95th percentile is 1.0 percent. In the case we ran, the accuracy was 27.6 percent. Certainly, for a less spread distribution, the accuracy would be less. However, even for a random function with an error factor of 3, the theoretical accuracy of Monte Carlo simulation (see formula 23) is about 4 percent, which is still greater than 1 percent accuracy claimed by SAMPLE. Table 2 Comparison of Simulated and Exact Analytical Parameters (Sample Size 1200) Parameter Simulated Value Exact Value Ratio of Simulated and Exact Values Relative Error (%) Lower Limit (5th percentile) x 10" x 10" Median (50th percentile) x 10~ x 10~ Upper Limit (95th percentile) x 10" x 10" Mean x 10" x 10" 3 Standard Deviation x

3 ACCURACY OF MONTE CARLO SIMULATED DISTRIBUTIONS Most applications of the Monte Carlo method are connected with calculation using simulations of some multidimensional integrals. This problem is equivalent to calculation of the expectation of some random number. Tho evaluation of error can be done with the central limit theorem [2]: x-x < My) a (1) where x is a Monte Carlo estimate of the exact value x ; o is standard p o deviation of the simulated random function; n is a number of simulations; and A(y) is a parameter depending on the chosen confidence y. For example, for confidence y=0.9 (90 percent confidence interval) the parameter y(0.9)= In the PRA application of the Monte Carlo method we have a different problem. We have to evaluate the absolute error Ax or relative error 6 of some random value x which is a specified percentile of the simulated distribution f(x). As far as we know, this problem was not covered in the current literature. Consider the Monte Carlo simulation of random number x with known density function f(x) and cumulative probability F(x). For this purpose we generate uniformly distributed random number y in the range: 0 < y < 1 (2) Let n be a number of trials in one simulation set with some initial seed number and y. (j=l, 2,... n) are sorted in ascending order generated random J numbers. Then determine numbers x. (j=l, 2,... n) according to the formula: x. y. = F(x.) = f(x)dx (3) or <V (4) where F (y) is an inverse function of the function F(x). For a large number of simulations n, each number y. is equal approximately: I n (5)

4 Denote p. = - x 100 (7) A number x. corresponding to y. is approximately a f.- fractile or a p.- percentile for the density function f(x). The closer the number y. is to the f., the better estimate for f.-fractile the number x. is. The deviation of the numbers y. from the f. is quite random. different sets of simulations with different seed numbers we get different sets of the y. scattered around the corresponding f.. If we take What is the probability that some randomly selected number y appears to be an f.-fractile? If y is an f.-fractile, it has to satisfy the condition: y ± < y < y i+1 (8) Therefore, i out of n simulated numbers have to be in the interval [0, y]; hence, the probability density g(y) that y is f.-fractile is equal to the probability density that i out of n simulated numbers y. are in the interval [0, y]. The last probability density is given by ieta distribution [2]: 8(y) = i; The mode m(y), mean E(y), and variance V(y) of the distribution (9) are respectively: m(y) = \ = f i (10) (y) = ig (ID V(y) = W ) (n-i+d (12) If n -* we have an asymtotic expression for mean and variance: E(y) = f i (13) V(y) = \ i i (1-f^ (14) For the large n, the distribution (9) tends to be a normal distribution with mean f. and standard deviation a. defined by formula: I l a. = V f U-f.) (15) I, /» I i

5 Therefore, the one standard deviation confidence interval for y can be determined as: VTT The absolute error Ax of estimating f.-fractile (or p.-percentile) according to formula (3) can be expressed in the form: Ax. = Ay i ix.. (16) (17) where absolute error Ay can be evaluated with formula (16). expression for Ax is: Thus, the final (18) Formula (18) assumes that we have a perfect and unbiased generator of random numbers and uniformly accurate algorithm for the calculation of the inverse function F (y), otherwise the error could exceed the expression (18). Formula (18) is applicable in the case when random value x is a function of several random arguments x,, x 0,...x and the density function f(x) is found from the Monte Carlo simulation technique. 4. SIMULATION TEST FOR LOGNORMAL DISTRIBUTION Density function f(x) for the lognormal distribution takes form: f(x) = yjlno exp (In x-l 2a 2 (19) where (J and a are logarithmic mean and standard deviations. Therefore, a relative error 6 = for the lognormal distribution at p.-percentile is: 6. = I a Vn" exp (In x i ~(j) 21 (20) All factors depending on percentile for median, lower, and upper limits are given in Table 3.

6 Table 3 Factors Depending on Percentile in the Formula for Relative Error Factor Lower Limit (5th percentile) Median (50th percentile Upper Limit (25th percentile) f. 1 In \/ J Ld-f.) ex] X. 1 > "(In x. -p) 2a M-1.645a M [j+1.645a Using data from Table 3, we can readily find the expression for relative error at lower limit, median, and upper limit: lower a (21) 5,-.median a (22) 6.upper (23) To check our theoretical formulas we performed several simulation tests. First, we tested formula (15). For this purpose we organized two series of tests. The first series consisted of 25 runs with 1,000 trials per run and a different initial seed number for every run. The second series consisted of 25 runs with 10,000 trials per run. Then we calculated the standard deviation for median, lower, and upper limits for every series and compared them with theoretical formula (15). The results of this comparison are shown in Table 4. The relative difference between theoretical and empirical results is in the range of 1 to 2 percent. We then tested formulas (21) through (23) for a lognormal distribution. Our test consisted of three series with 10 runs each and a number of trials equal to 100, 1,000 nnd 10,000, respectively. Each run started with a different seed,number. In our test, we generated a lognormal distribution with median 10 and an error factor 100,000 (JJ= , 0= ). The comparison of empirical and theoretical relative errors is given in Table 5.

7 Table 4 Comparison of Empirical and Theoretical Standard Deviations for Uniform Distribution Lower Limit (5%) Median (50%) Upper Limit (95%) n Empirical Theoretical Empirical Theoretical Empirical Theoretical 1, xl0" xlO" 3 1.6lxlO" xlO" xlO" xl0" 3 10, xl0" xl0" xl0" xlO~ xlO" 3 2.I8xl0" 3 Table 5 Comparison of Empirical and Theoretical Relative Errors for Lognormal Distribution Lower Limit (5%) Median (50%) Upper Limit (95%) n Empirical Theoretical Empirical Theoretical Empirical Theoreti cal , , The results of comparison show that we can use formulas (21) through (23) for evaluation of relative error of lognormal and similar distributions. 5. TIME COMPLEXITY AND COST OF A COMPUTER RUN FOR DISTRIBUTION SIMULATION To reach 1 percent accuracy at the upper limit for a lognormal distribution with a parameter a of approximately 5, one million simulations are needed. This is true for an arbitrary distribution simulated by computer code. To assess the feasibility of the Monte Carlo method, we have to estimate the computing time and the cost of a computer run.

8 The computing time of Monte Carlo simulation programs is proportional to the time complexity T(n) of its algorithm [3] where n is a number of simulations The time complexity of the simulation computer code is limited by its sorting subroutine. The best sorting subroutine has time complexity T(n) given by [3]: T(n) - n log n (24) Usually the cost of a computer run C should be estimated according to the progressive cost rate. T(n) Figure 1 RELATIONSHIP BETWEEN THE COST OF A RUN AND COMPUTING TIME The typical relationship between the cost of run and computing time on the arbitrary scale is shown in Figure 1. This relationship can be fitted by a smooth curve: C = A [T(n)] k (25) Therefore, the asymptotic cost as a function of the number of simulations is: C ~ (n log n)' (26) where k > 1 (27)

9 The required number of simulations n depends on relative error 6. iognormal distribution: For n ~?-;. (28) Putting (28) into (24) and (26) we obtain the asymptotic expressions for time complexity T and costs of computer run C depend on the relative error 6 and logarithmic standard deviation a: MIL (29) 2k (30) Therefore, to increase accuracy ten times, we have to increase the computing time more than 100 times, according to formula (29). The cost increase will be even more significant. The question is whether there are methods of increasing the accuracy of the Monte Carlo method without increasing computation time and the cost of the run. For applications considered at the beginning of Section 3, which can be reduced to the evaluation of a random-sample average, there are three methods of increasing accuracy: stratified sampling, correlated sampling, and importance sampling. All three methods belong to the so called "variancereduction" technique [2], According to formula (1) if we substitute one sample with another having the same mean but less standard deviation a n we can reduce error while keeping the same computation time. These variance-reduction methods cannot significantly save the computing time and, unfortunately, they are not applicable to the problem of simulation of distributions because the variance of distributions must be kept intact. With this in mind, a new method of increasing accuracy will be considered below. 6. IMPROVED MONTE CARLO SIMULATION METHOD The conventional approach of estimating the p.-percentile of some simulated distribution consists of determining the number x. from the sorted array x. (j=l,2,...n) of the random variable x with index number i defined from formula (7). Therefore, only one value out of n random numbers is used to determine the p.-percentile. In this case, the only way to increase accuracy is to increase the number of simulations n; and the cost of this method is very high. Meanwhile, all points x. (j=l,2,... n) representing some distribution contain the information of p.-percentile. Below, we consider the special method of extracting this information.

10 Let x. (j=l,2,... n) be a sorted set of random numbers in ascending order and f (x; re a... a.) be a test function with k fitted parameters a a,... a to fit the distribution of a random variable x. ff n is sufficiently large, any v.ilue x. is an approximate estimate of p.-percentile. Hence, the numbers y. (a., a 0,... a.) defined as f y. (a r u 2,.-a n ) = I f (x; a 1> a,,,... a fc ) dx (31) are supposed to be close to f. given by equation (6) if the function f(x; a a... a.) and parameters a,, a L o,... a, are chosen appropriately. Consider the parameters 4.- ( a i > a y > a u) given by formula: y (a, a... a ) - f i (a v a 2,... re k ) = -! ^ 1 ^ i (32) i where a. are determined by equation (15). As previously explained, fcr a large number n, every parameter 4- is a normal random variable with ^ero mean and a standard deviation equal to unity. Introduce two random fui ctions F, and according to formulas: a 2>... a k ) =i ^. (a r a 2,... a fc ) (33) F (a, a... a )= ) k (a a... or )] (34) If lae test function f(x; a,, a,... a.) is a good fit, the function F, has a, ^rmal distribution witn zero mean and a standard deviation equal to 1/ Jn~and the function F_ has a chi-square distribution with a parameter equal to n. Therefore, for a good fitting function f(x; a., a,... a,) two conditions have to be satisfied: nf 1 2 < 1 (35) r 2 " '

11 These conditions mean that both functions F, and F deviate from their means by no more than one standard deviation, which is acceptable. Thus, to find the best fit for a function f(x; u, a,...'. a^) we have to minimize simultaneously the left sides of equations (35) and (36). However, it is easier analytically to minimize the expression G(a., a?,... a,) If the inequality G < 1 is met, the corresponding approximation can be considered adequate. The approximation which meets the inequality G < 0.5 (38) is considered as good because both conditions (35) and (36) will be met automatically. Assume that we found the best fitting function f(x; Of-,, a»,... a.) giving the minimum to the expression (37) and satisfying conditions (35) and (36) or one stronger condition (38). Using this function we can find corrected percentiles x. of the distribution in question as: x. f (x; a v a 2,... ff fc ) dx = f ± (39) For evaluation of p.-percentile, instead of random number x. with standard deviation given^by equation (18),_we use its mean x.. Because the number of points participating in finding x. is equal to n, the standard deviation of x. and corresponding absolute error Ax. will be reduced by Yn" times: Ax. = \, / - (40) I f(x.) n Relative errors for lognormal and similar distributions will be: a lower n (41) 6 A. =h2s33o (42) median n 6 = (43) upper n

12 and asymtotic expressions for time complexity T and cost of computer run C will take the form: (44) (45) The additional algorithm when minimizing the function G has time complexity T(n) ~ n and cannot change the asymtotic estimate given by equation (44). Hence, the proposed method can reduce the computing time and the cost of computer run relatively to the conventional Monte Carlo method at the same level of accuracy. 7. LIMITATIONS The improved Monte Carlo simulation method is based on the assumption that we have a perfect unbiased generator of random numbers. It means that uniform distribution simulated with such a generator satisfies the conditions (35) and (36). Unfortunately, most existing generators fail this test. In the case of a biased generator of random numbers, the final simulated distribution is also biased. Thus, application of the above technique improves the precision rather than the accuracy of the simulation. This technique does, however, improve the result of simulation but the degree of improvement is less than that predicted by formulas (41) through (43). We have to be cautious of "the best analytical fit" if we use the biased generator of random numbers. Also, we have to recognize that conditions (35) and (36) can be either conflicting or not achievable at all if we fail to chose the appropriate test function f(x; a,, a,... a ) or an unbiased generator of random numbers. Therefore, from a practical point of view, it is better to minimize the expression in the left side of inequality (36) without the strong conditions of (35) and (36). This "weaker" goal will always be achievable. ACKNOWLEDGEMENTS The author wishes to thank Dr. M. Reier for testing the SAMPLE code and useful discussions of some aspects of this paper. REFERENCES 1. Reactor Safety Study, WASH-.1400, Appendix II, pp. II-64-II-74, Oct Korn, G. A., Korn, J. M., "Mathematical Handbook for Scientists and Engineers", McGraw-Hill Book Company, New York, Aho, A. V., Hopcroft, J. K., Ullman, J. D., "The Design and Analysis of Computer Algorithms", Addisow-Wesley Publishing Company, Massachusetts, 1974.