Generating pseudo Normal Numbers : Ziggurat, Box Muller and R default rnorm method

Size: px

Start display at page:

Download "Generating pseudo Normal Numbers : Ziggurat, Box Muller and R default rnorm method"

May Franklin
5 years ago
Views:

1 Generating pseudo Normal Numbers : Ziggurat, Box Muller and R default rnorm method 1 Introduction Mriganka Aulia ISI There is many methods for generating pseudo Normal Random Numbers. Here we focus on only three of these methods i) Ziggurat ii) Box Muller iii) R default rorm function The main idea of Ziggurat method was developed by George Marsaglia and others in 1960s. The aim of this method is to covering the whole density by some known density and draw a random point from the second one and declare it as a random point from the first if it obeys some criteria. And in 1980s Marsaglia and Tsang have developed the ziggurat method for sampling from decreasing densities by expressing the whole density as a mixture of small rectangles and draw a random uniform point from a randomly chosen rectangle. The Box Muller method was introduced by George Edward Pelham Box and Mervin Edgar Muller in But the method was first mentioned by Paley and Wiener in Box Muller transformation can transform two independent uniform number into two independent Normal number. And R default rnorm function is solely based on approximating the Normal cdf by some polynomial. This is done by Ross Ihaka in And this is based on two papers AS 111 (C) 1977 Royal Statistical Society and AS 241 (C) 1988 Royal Statistical Society.

2 2 Developing Basic Ideas 2.1 Ziggurat Method Rejection Method : This method is depend on the basic rejection principle. Let C be the set of all under a plot of the curve y = f(x) with finite area and let Z be the set of points containing C i.e. Z C. Then the idea is to choose a random point from Z until we get one that falls in C and then return it as the required variate. Here the density of such an x is Normal and we take a Uniform density as a covering density. We draw a random number from that uniform density until we get one point that falls under the Normal curve and we return it as a Normal random number. Partitioning the Normal Curve : Now we can increase the efficiency of the rejection procedure by minimizing the area(c)/area(z). And to achieve this, instead of covering the whole Normal curve by a Uniform pdf we cover the whole pdf by various small rectangles and do the rejection procedure on the small rectangle instead of the big one. Now to make the rectangles equally probable, we make the rectangles in such way that they have equal area. For simplicity let we divide the pdf into 7 rectangles with a base partition, in total 8 partition. let x 1 be the rightmost x coordinate of the rectangles and we have the partition like x 1 > x 2 > x 3 > x 4 > x 5 > x 6 > x 7 > x 8 = 0.

3 Drawing a Random Number from the Partitions : We first select a partition randomly. Let we choose ith partition When it s not the base partition ( i > 1) Now we draw a random number u form Uniform( 0, Xi ) and check i) If u < Xi+1, then return u as a Normal number. ii) Else we draw a bivariate point form that rectangle with x coordinate as u i.e. we generate another random u 1 from Uniform( (Xi), (Xi + 1) ) where (x) is the Normal pdf and check if (u, (Xi) + u 1 ) is under the Normal curve, (Xi) + u 1 < (u), then return u as a Normal number. iii) If both the cases fail then we again choose a new partition.

4 If it is base partition, i = 1 We generate a random number u 2 from U(0, A), A be the common area of the partition. i) If u 2 < x 1 (x 1) then return some v ~ U(0, x 1 (x 1) ) ii) else generate exponential random number from the tail, let V 1 and V 2 be two random number from U(0, 1). Then Marsaglia suggests a compact algorithm: 1. Let x = ln(v 1)/x Let y = ln(v 2). 3. If 2y > x 2, return x + x Otherwise, go back to step 1.

5 2.2 Box Muller Method : Let u 1 ~ U(0,1) and u 2 ~ U(0,1), independently. Then, R 2 = - 2ln(u 1) ~ exp(mean = 2) by probability integral transformation and θ = 2πu 2 ~ U(0, 2π ), independently. So f, (r, θ) = e, r 2 > 0 and 0 < θ < 2π Let X 1 = R cos(θ) and X 2 = R sin(θ) Jacobian of this transformation is J = 2X 2X = 2 f, (x, x ) = e f (x ) = f, (x, x ) dx = f (x ) = f, (x, x ) dx =, < x < and < x < π e x 2, < x < 2 1 2π e x 2, < x < So, X 1 ~ N(0, 1) and X 2 ~ N(0, 1) independently. 2.3 R default rnorm method : The R default rnorm function depend on R default qnorm function which returns the value of a quantile for a given probability. First draw a random number u from U(0, 1) and then return x = qnorm(u), which is a required random Normal number. Now qnorm works by approximating the normal cdf by polynomial given by Algorithm AS 111 by J. D. Beasley and S. G. Springer and Algorithm AS 241 by Michael J. Wichura. Let we have 0 < p < 1 and we want the corresponding quantile.

6 0 e e We partition the (0,1) length into three parts i) When < p < Compute q = p 0.425, so q < then take r = q 2 and required quantile. ii) When e -25 < p < or < p < 1 - e -25 i.e. min(p,q) < and r1 = log(min(p, q)) 5 Then return zp = ± E(r 1 ), the sign is determined by q 0 or q < 0, E and F both F(r 1 ) are 7th degree polynomial. iii) When 0< p < e -25 or 1 - e -25 < p < 1 i.e. if r1 > 5 Then return z p = ± G(r 1 ) H(r 1 ) of degree 7., sign is + or according to q 0 or q < 0, G and H also

7 3 Implementation : a. Ziggurat Method : Setting up Necessary Things : Finding the x coordinates of the Partition points : We have to form a partition in such way that the partition exactly covers up the whole Normal cdf curve with each partition having equal area. Let x 1 be the rightmost point then common area A = area of the base partition. Then A = area of the base rectangle + area of the tail = x (x ) + ( 1 Φ(x ) ), be the Normal pdf and Φ be the Normal cdf. Now we form a rectangle on top it with base x 1 and height y 1 = and form x 2 where it cuts the Normal curve i.e. x 2 = (y ) So yi = and y = y, x i+1 = (y) until y and let t be the total number rectangles drawn upto y We have to find some x 1 such that e 0.. So area left at the top p = 0.5 t*a.

8 The below R code is giving the area left at the top for a given leftmost partition point : area <- function(r) # r <-rightmost point of the partition A <- (r*dnorm(r)) pnorm(r) # starting with common area A x <- NA x[1] <- r i <- 1 y <- dnorm(x[1]) while(y <= dnorm(0)) x[i] <- sqrt(-2 * log(sqrt(2*pi)*y)) if(x[i]!= 0) y <- dnorm(x[i]) + (A/x[i]) else y <- dnorm(0) + 1 i <- i+1 j <- i-1 p <- pnorm(x[j]) (x[j]*dnorm(x[j])) # area left at the top x[j+1] <- j x[j+2] <- A x[j+3] <- p # no of partition # common area of each partition # area actually left at the top x[j+4] <- (j * A) p # Total area where rejection can happen x[j+5] < ((x[j+4]/(j * A)) * 100) # Percentage of acceptance return(x[j+3])

9 Then, we take some sequence from 3.5 to 4.2 and plot the upper area function for each of the value of the sequence and then use which.min function to determine the value where the minimum occurred. # ploting and finding the min value of area for given a seq of partition p oint from 3.5 to 4.2 r <- seq(from = 3.5,to = 4.2,length = 1000) plot(sapply(r, area), ylab = "Left Upper Area") abline(h=0) k <-r[which.min(sapply(r, area))] k [1] area(k) [1] e-08 Then we improve the value by choosing a sequence from to # improving the min by choosing the seq in to r <- seq(from = , to = , length = 1000) k <- r[which.min(sapply(r,area))] k [1] area(k) [1] e-11 Then after doing some trial and error we find r < area(r) [1] e-14 So we find the right most point as and by choosing this value as x 1 the area left is *

10 And we got the partition by : area <- function(r) A <- (r*dnorm(r)) pnorm(r) x <- NA x[1] <- r i <- 1 y <- dnorm(x[1]) while(y <= dnorm(0)) x[i] <- sqrt(-2 * log(sqrt(2*pi)*y)) if(x[i]!= 0) y <- dnorm(x[i]) + (A/x[i]) else y <- dnorm(0) + 1 i <- i+1 j <- i-1 p <- pnorm(x[j]) (x[j]*dnorm(x[j])) x[j+1] <- j x[j+2] <- A x[j+3] <- p # no of partition # common area of each partition # area actually left at the top x[j+4] <- (j * A) p # Total area where rejection can happen x[j+5] < ((x[j+4]/(j * A)) * 100) # Percentage of acceptance return(x) l <- area( ) l[257] [1] 256 We got total 256 partition with x 1 = and

11 l[261] [1] Total acceptance probability of any number drawn from those rectangles is %. Simplifying the Algorithm : The Ziggurat Algorithm is 1. Choose a random layer 1 i Let x = U 0x i where U 0 ~ U(-1, 1). 3. If x < xi+1, return x. 4. If i = 1, generate a point from the tail using the fallback algorithm. 5. Let y = y i + U1(y i+1 y i) where y i = (x ). 6. If y < (x), return x. 7. Otherwise, choose new random numbers and go back to step 1. For i = 1, Generate u 2 = U*A, U ~ U(0, 1) and check if u 2 < x 1 (x ), then return u ~ U(- x 1 (x ), x1 (x ) ). Oherwise follow, The fallback algorithm 1. Let x = ln(u1)/x1 where U 1 ~ U(0, 1). 2. Let y = ln(u2) where U 2 ~ U (0, 1). 3. If 2y > x 2, return x + x1. 4. Otherwise, go back to step 1. Let i 1. We convert the whole x i s and y i s into another p, q and d s for the sake of runtime p[i] = x[i-1] r[i] <- x[i]/x[i-1] d[i] <- 2π * (x[i]) and i = 1 p[1] = x[1] r[1] = x[1] * (x[1]) / A d[1] = 2π (x[1]) So, the case x = U 0xi and u = A*U both can be converted into x = upi, u be any random sample from U(0, 1).

12 and x < xi+1 and u < x 1 (x ) can be converted into u < r[i] and the checking y = y i + U1(y i+1 y i) < (x) can be done by checking U1 (d[i+1] d[i]) > (e d[i] ) Function for generating the new p[i], q[i]and d[i] s : pr1 <- function(x,j) p <- NA r <- NA d <- NA p[1] <- x[1] r[1] <- x[1]*dnorm(x[1]) / x[j+2] d[1] <- sqrt(2*pi) * (dnorm(x[1])) for(i in seq(2,j)) p[i] <- x[i-1] r[i] <- x[i]/x[i-1] d[i] <- sqrt(2*pi) * (dnorm(x[i])) p[j+1] <- x[j] c <- list(p,r,d) return(c) RCPP coding for the Ziggurat function : #include <Rcpp.h> #include <math.h> using namespace Rcpp; // [[Rcpp::export]] NumericVector c_zrnorm1 (NumericVector p, NumericVector r, NumericVector d, int n) NumericVector y(n) ; int e = 0 ; while(e < n)

13 int j = 0 ; while (j < 1) int i = (int)(256 * runif(1)[0]) ; double u = (2* runif(1)[0]) - 1 ; double x = p[i] * u ; if(abs(u) < r[i]) y[e]= x ; j = j+1 ; else if(i == 0) double y1 = -log(runif(1)[0])/p[0] ; double y2 = -log(runif(1)[0]) ; while(((y2 +y2) > (y1 * y1))) y1 = -log(runif(1)[0])/p[1] ; y2 = -log(runif(1)[0]) ; if(u >= 0) y[e] = (p[1] + y1) ; else y[e] = -(p[1] + y1); j = j+1 ; else if(((d[i] - d[i-1]) * runif(1)[0]) < (exp(-pow(x,2)/2) - d[i-1])) y[e] = x ; j = j+1 ; e = e + 1 ; return(y) ;

14 Main zrnorm1 function in R : library(rcpp) sourcecpp("zrnorm1.cpp") zrnorm1 <- function(size,mean,sd) mean + (sd * c_zrnorm1(p2,r2,d2,size)) zrnorm1(10,5,3) [1]

15 b. Box Muller method : We generate two independent u 1 and u 2 from U(0, 1) distribution and form X 1 and X 2 by Box Muller transformation and return them. i) If sample size n = even = 2k then we return X 1 and X 2 k times ii) If sample size n = odd = 2k + 1 then return X 1 and X 2 k times and generate another X 1 and X 2 and return only X 1. RCPP coding of Box Muller method : #include <Rcpp.h> #include <math.h> using namespace Rcpp; // [[Rcpp::export]] NumericVector boxmuller1(int size) NumericVector x(size) ; int i = (int)(size * 0.5); NumericVector u = runif(i); NumericVector v = runif(i); int j = 0 ; while(j < i) x[j] = sqrt(-2 * log(u[j]))*cos( * v[j]); x[i+j] = sqrt(-2 * log(u[j]))*sin( * v[j]); j = j+1 ; if(size%2 == 1) x[size-1] = sqrt(-2*log(runif(1)[0]))*cos(2* * runif(1)[0]); return(x);

16 Main boxmuller1 function in R : library(rcpp) sourcecpp("boxmuller.cpp") boxmuller1 <- function(size,mean,sd) mean + (sd * c_boxmuller1(size)) boxmuller1(10,5,3) [1]

17 c. Default rnorm function : The function rnorm uses the function norm_rand which is function for generating random normal num But rnorm use Inversion method as default. C code for rnorm( 1998 Ross Ihaka) : #include "nmath.h" double rnorm(double mu, double sigma) if (ISNAN(mu)!R_FINITE(sigma) sigma < 0.) ML_ERR_return_NAN; if (sigma == 0.!R_FINITE(mu)) return mu; /* includes mu = +/- Inf with finite sigma */ else return mu + sigma * norm_rand(); INVERSON method(1998 Ross Ihaka) : This method uses the qnorm function which returns the quantiles. It first i) Generate a u 1 ~ U(0, 1) and u 2 ~ U(0, 1) independently ii) Then take p = to have a high precision and use p as probability and return the qu corresponding to p by qnorm4 function. C Code for INVERSION Method : case INVERSION: #define BIG /* 2^27 */ /* unif_rand() alone is not of high enough precision */ u1 = unif_rand(); u1 = (int)(big*u1) + unif_rand(); return qnorm5(u1/big, 0.0, 1.0, 1, 0); C code for qnorm5 function (1998 Ross Ihaka) : #include "nmath.h" #include "dpq.h" double qnorm5(double p, double mu, double sigma, int lower_tail, int log_p) double p_, q, r, val;

18 #ifdef IEEE_754 if (ISNAN(p) ISNAN(mu) ISNAN(sigma)) return p + mu + sigma; #endif R_Q_P01_boundaries(p, ML_NEGINF, ML_POSINF); if(sigma < 0) if(sigma == 0) ML_ERR_return_NAN; return mu; p_ = R_DT_qIv(p);/* real lower_tail prob. p */ q = p_ - 0.5; #ifdef DEBUG_qnorm REprintf("qnorm(p=%10.7g, m=%g, s=%g, l.t.= %d, log= %d): q = %g\n", p,mu,sigma, lower_tail, log_p, q); #endif /*-- use AS */ /* double ppnd16_(double *p, long *ifault)*/ /* ALGORITHM AS241 APPL. STATIST. (1988) VOL. 37, NO. 3 Produces the normal deviate Z corresponding to a given lower tail area of P; Z is accurate to about 1 part in 10**16. */ (original fortran code used PARAMETER(..) for the coefficients and provided hash codes for checking them...) if (fabs(q) <=.425) /* <= p <= */ r = q * q; val = q * (((((((r * ) * r ) * r ) * r ) * r ) * r ) * r ) / (((((((r * ) * r ) * r ) * r ) * r ) * r ) * r + 1.); else /* closer than from 0,1 boundary */ /* r = min(p, 1-p) < */ if (q > 0) r = R_DT_CIv(p);/* 1-p */ else r = p_;/* = R_DT_Iv(p) ^= p */ r = sqrt(- ((log_p && ((lower_tail && q <= 0) (!lower_tail && q > 0)))? p : /* else */ log(r))); /* r = sqrt(-log(r)) <==> min(p, 1-p) = exp( - r^2 ) */ #ifdef DEBUG_qnorm REprintf("\t close to 0 or 1: r = %7g\n", r); #endif if (r <= 5.) /* <==> min(p,1-p) >= exp(-25) ~= e-11 */ r += -1.6; val = (((((((r * e-4 +

19 ) * r ) * r ) * r ) * r ) * r ) * r ) / (((((((r * e e-4) * r ) * r ) * r ) * r ) * r ) * r + 1.); else /* very close to 0 or 1 */ r += -5.; val = (((((((r * e e-5) * r ) * r ) * r ) * r ) * r ) * r ) / (((((((r * e e-7)* r e-5) * r e-4) * r ) * r ) * r ) * r + 1.); if(q < 0.0) val = -val; /* return (q >= 0.)? r : -r ;*/ return mu + sigma * val;

20 4 Normality Checking : 4.1 Shapiro Wilk Test Sample size = 1000 : Ziggurat (zrnorm1 ) : length(which(replicate(1000,shapiro.test(zrnorm1(1000,0,1))$p.value) < 0.05))/ 1000 [1] Box Muller (boxmuller1) : length(which(replicate(1000,shapiro.test(boxmuller1(1000,0,1))$p.value) < 0.05))/1000 [1] Sample size = 5000 : zrnorm1 : length(which(replicate(1000,shapiro.test(zrnorm1(5000,0,1))$p.value) < 0.05))/ 1000 [1] boxmuller1 : length(which(replicate(1000,shapiro.test(boxmuller1(5000,0,1))$p.value) < 0.05))/1000 [1] So in all cases zrnorm1 and boxmuller1 both are accepted by Shapiro Wilk Test. 4.2 Kolmogorov Smirnov test for Normality : Sample size = zrnorm1 : library(nortest) length(which(replicate(1000,lillie.test(zrnorm1(10000,0,1))$p.value) < 0.05))/ 1000 [1] 0.05

21 boxmuller1 : length(which(replicate(1000,lillie.test(boxmuller1(10000,0,1))$p.value) < 0.05))/1000 [1] So both the functions are accepted by this test Sample size = zrnorm1 : length(which(replicate(1000,lillie.test(zrnorm1(50000,0,1))$p.value) < 0.05))/ 1000 [1] boxmuller1 : length(which(replicate(1000,lillie.test(boxmuller1(50000,0,1))$p.value) < 0.05))/1000 [1] So again it accepted by the test. So both functions are giving Normal Random Number. Next we test for randomness.

22 5 Test for Randomness : Wald-Wolfowitz Runs Test : 5.1 Sample Size = 5000 i. zrnorm1 : length(which(replicate(1000,runs.test(zrnorm1(5000,0,1))$p.value) < 0.05))/ 1000 [1] ii. boxmuller1 : length(which(replicate(1000,runs.test(boxmuller1(5000,0,1))$p.value) < 0.05))/ 1000 [1] Sample Size = zrnorm1 : length(which(replicate(1000,runs.test(zrnorm1(50000,0,1))$p.value) < 0.05))/ 1000 [1] boxmuller1 : length(which(replicate(1000,runs.test(boxmuller1(50000,0,1))$p.value) < 0.05))/ 1000 [1] So in all cases hypothesis is accepted that these two functions are generating random number.

23 6 Runtime Comparison : a. Number per Second : zrnorm1 : mean(replicate(10,system.time(zrnorm1(10^7,0,1))[1])) [1] (10^7) / [1] Boxmuller : mean(replicate(10,system.time(boxmuller1(10^7,0,1))[1])) [1] (10^7)/ [1] rnorm : mean(replicate(10,system.time(rnorm(10^7,0,1))[1])) [1] (10^7)/ [1] Method Random Number per second Ziggurat zrnorm1 Box Muller boxmuller1 rnorm So rnorm is faster than the other two functions.

24 a. Runtime for 10 8 sample case : i. Ziggurat : mean(replicate(5,system.time(zrnorm1(10^8,0,1))[1])) [1] ii. Box Muller : mean(replicate(5,system.time(boxmuller1(10^8,0,1))[1])) [1] iii. rnorm : mean(replicate(5,system.time(rnorm(10^8,0,1))[1])) [1] Method Runtime for 10 8 sample (in sec.) Ziggurat zrnorm1 Box Muller boxmuller1 rnorm b. Drawing a curve showing runtime for the three functions : time1 <- function(size) return(system.time(zrnorm1(size,0,1))[1]) time2 <- function(size) return(system.time(boxmuller1(size,0,1))[1]) time3 <- function(size) return(system.time(rnorm(size))[1])

25 i. Draw a curve for sample of size to by : t <- seq(10^6, 2e+07, by =10^6) plot(sapply(t,time1), type = "l", col = "blue", main = "Runtime Comaprison", xlab = "Number of samples (in 10^6)", ylab = "Runtime", asp = 1 ) lines(sapply(t,time2), type = "l", col = "red", asp = 1) lines(sapply(t,time3), type = "l", col = "green", asp = 1) abline(h = 0, col = "orange") Red : zrnorm1 Blue : boxmuller1 Green : rnorm

26 Sample of size to by : t <- seq(10^5, 10^6, by = 10^5) plot(sapply(t,time1), type = "l", col = "blue", main = "Runtime Comaprison", xlab = "Number of samples (in 10^6)", ylab = "Runtime") lines(sapply(t,time2), type = "l", col = "red") lines(sapply(t,time3), type = "l", col = "green") Red : Ziggurat Blue : Box Muller Green : rnorm

27 References 1. Rejection Samping : 2. Box Muller Transformation : 3. Ziggurat Algorithm : 4. Paper by Marsaglia and Tsang : 5. Source code for rnorm : 6. Source code for Inversion method : 7. Source code for qnorm : 8. Algorithm AS 111 by Beasley and Springer : 9. Algorithm AS 241 by Michael Wichura : Shapiro Wilk Test : Kolmogorov Smirnov Normality Test : Wald Wolfowitz runs test :

STATISTICAL LABORATORY, April 30th, 2010 BIVARIATE PROBABILITY DISTRIBUTIONS

STATISTICAL LABORATORY, April 3th, 21 BIVARIATE PROBABILITY DISTRIBUTIONS Mario Romanazzi 1 MULTINOMIAL DISTRIBUTION Ex1 Three players play 1 independent rounds of a game, and each player has probability