Sampling from distributions

Transcription

1 Sampling from distributions December 17, Sampling from distributions Now that we are able to sample equally distributed (pseudo-)random numbers in the interval [1, 0), we are now able to sample random numbers from arbitrary distributions. 1.1 Setup In [1]: %matplotlib inline import numpy as np import scipy as sp import matplotlib.pyplot as plt 1.2 Rejection sampling The most simple method to generate random numbers from a distribution is called rejection sampling. This method is generally easy to implement, but can be very slow depending on the sampled distribution. Further info can be found at Wikipedia. Assume we want to sample data from a probability density function f. Assume further we are able to genrate random numbers according to a distribution g. Also, we have a bound M, so that f(x) < Mg(x) x. Then we can sample random numbers distributed like f: 1. Sample a random number u [0, 1) 2. Sample a random number x from g(x) 3. Calculate = f(x) Mg(x) u If 0, x is distributed according to f, and we return the random number x. Else, we reject the result and repeat the process. For example, we can use equally distributed random numbers in an interval [a, b], and select an M so that g(x) = 1 x [a, b). In practice, we can just set g(x) = M = 1. So let s write the code to implement what we described above: In [2]: def gauss(x, mu=0., sigma=1.): return 1./(sigma * np.sqrt(2.*numpy.pi)) * np.exp(-0.5 * ( 1.*(x-mu) / sigma)**2) def rejection_sampler(func, a=0., b=1., M=1., *funcargs, **funckwargs): delta = -1 x = 0 while not delta >= 0: # generate uniformly distributed random number in the interval [a, b) x = (b-a) * np.random.random() + a # generate uniformly distributed random number in the interval [0,1] u = np.random.random() * M # calculate delta delta = 1.*func(x, *funcargs, **funckwargs) - u return x def rejection_sampler_example(func, a=0., b=1., M=1., *funcargs, **funckwargs): 1

2 delta = -1 x = 0 # generate uniformly distributed random number in the interval [a, b) x = (b-a) * np.random.random() + a # generate uniformly distributed random number in the interval [0,1] u = np.random.random() * M # calculate delta delta = 1.*func(x, *funcargs, **funckwargs) - u return x, u, delta >= 0 # limit the range of sampling # choosing larger intervals increases computing time due to more rejections x = numpy.arange(-20.,20.,0.01) a = x[0] b = x[-1] M = 0.6 # gaussian distribution does not exceed 0.6 r = numpy.array([rejection_sampler(gauss, a, b, M) for i in xrange(50000)]) r_e = numpy.array([rejection_sampler_example(gauss, a, b, M) for i in xrange(1000)]) # plot the histogram of the sampled numbers h, xb = np.histogram(r, bins=1000, density=true) plot(xb[:-1]+np.diff(xb), h, color= b, label= Rejection sampled normal distribution ); #plot(r_e[:,0][r_e[:,2]==0], r_e[:,1][r_e[:,2]==0],linestyle=, marker= x, color= y ) #plot(r_e[:,0][r_e[:,2]==1], r_e[:,1][r_e[:,2]==1],linestyle=, marker= o, color= r ) # overplot the calculated gauss x = numpy.arange(-6.,6.,0.01) y = gauss(x) plot(x,y, color= r, label= Normal distribution ); # prettify xlabel("x") ylabel("p") ylim(0., 0.7) xlim(-6.,6.) legend(); 2

3 Rejection sampled normal distribution Normal distribution 0.4 p x The algorithm works well, but as we can see is quite slow. The main reason is that on the wings the probability approaches zero fast, which means we reject a lot of random numbers. Every rejection means that we will have to call the function again. Thus, the runtime of this sampling method depends strongly on the similarity of the envelope distribution and the distribution to be sampled, and the chosen interval. 1.3 Inversion sampling The other widespread used method to sample random numbers is inversion sampling. Info on that can also be found at Wikipeda. Assume we want to take random samples distributed like a known continuous distribution function F (x). We have to do the following steps: 1. Normalize F F (x)dx = N, to calculate the probability density function (PDF) f(x) = F (x)/n 2. Calculate the cumulative distribution function (CDF) C(x) = x f(x )dx 3. Calculate the inverse CDF (icdf). This assumes that C(x) can be inverted (i.e. solved for x), so that we get a function X(U) 4. Sample a uniformly distributed random number u in the interval [0, 1) 5. Calculate x = X(u). The generated xs will be distributed according to the probability density function f(x) Let s implement the algorithm and inspect all the steps and generate random numbers according to an exponential distribution. The PDF of an exponential distribution is f(x) = λ exp ( λx), and it s defined on the interval [0, ) In [3]: x = np.arange(0,15,0.001) 3

4 In [4]: def expon(x, l=1.): return l * np.exp(-l * x) plot(x, expon(x), label="exponential distribution PDF"); legend(); Exponential distribution PDF The CDF of the exponential distribution is C(x) = 1 exp ( λx) In [5]: def expon_c(x, l=1.): return 1. - np.exp(-l * x) plot(x, expon_c(x), label="exponential distribution CDF"); legend(); 4

5 Exponential distribution CDF We can see nicely that the cumulative distribution function does not exceed 1. We can already see that if we invert the CDF C(x) and sample values on the y-axis, that we can generate the values on the x-axis. The icdf is X(U) = 1 λ ln (1 U). In [6]: def expon_ic(u, l=1.): return -1./l * np.log(1. - U) u = np.arange(0., 1., 0.001) plot(u, expon_ic(u), label="exponential distribution icdf"); legend(); 5

6 7 6 Exponential distribution icdf We can now write our random number generator, which creates numbers according to the exponential distribution, and try it out. In [7]: def rand_expon(l=1.): u = np.random.random() return expon_ic(u, l) r = np.array([rand_expon() for i in xrange(200000)]) h, xb = np.histogram(r, bins=1000, density=true) plot(xb[:-1]+np.diff(xb), h, color= b, label= Inversion sampled exponential distribution ); plot(x, expon(x), color= r, label= Exponential distribution ); legend(); 6

7 Inversion sampled exponential distribution Exponential distribution Let s compare the runtime of the inversion sampling method, to the runtime of the rejection sampling method. We can already see that it is very impractical to generate random numbers up to using rejection sampling. We compare both methods for different interval lengths. In [8]: import time b= 5.0 def rand_expon_interval(a=0, b=np.inf, l=1.): r = -1 while not (a <= r < b): r = rand_expon(l) return r for b in np.linspace(5., 25., 5.): print "b=", b a=0. start = time.time() inv = [rand_expon_interval(a,b) for i in xrange(20000)] end = time.time() print "Time to generate numbers using inversion sampling: ", end-start start = time.time() inv = [rejection_sampler(expon, a=a, b=b, M=1.0001) for i in xrange(20000)] end = time.time() print "Time to generate numbers using rejection sampling: ", end-start print 7

8 Time to generate numbers using inversion sampling: Time to generate numbers using rejection sampling: b= 10.0 Time to generate numbers using inversion sampling: Time to generate numbers using rejection sampling: b= 15.0 Time to generate numbers using inversion sampling: Time to generate numbers using rejection sampling: b= 20.0 Time to generate numbers using inversion sampling: Time to generate numbers using rejection sampling: b= 25.0 Time to generate numbers using inversion sampling: Time to generate numbers using rejection sampling: We can see that this method is much faster than rejection sampling, because the exponential function drops off very fast Inversion sampling of discrete (measured) distributions In case we only have access to a discrete representation of a distribution, for example data measured by an instrument, we can also perform inversion sampling, though in a discrete fashion. The steps are basically the same as above, we just replace the analytical methods with numerical ones. Assume we have measured data which is distributed according to a distribution function F (x), in discrete bins x 0, x 1,... x n 1. Normalize F (x), N = N i=0 F (x i), then calculate the probability density function f(x) = F (x) N 2. Determine the CDF C(x) as cumulative sum of f(x): C(x) = x i=0 f(x i) 3. Sample a uniform random number u in [0, 1). 4. Sort u into C(x), and store the index t, i.e. calculate the histogram of all us. 5. The histogram generated in step 4 is distributed like F. Let s implement this now. We have already sampled values form the normal distribution, so we now histogram those values to generate an artifical measurement. In [9]: r = [np.random.randn(2000)] bins = np.linspace(-3,3,13) h, xb = np.histogram(r, bins=bins) x = xb[:-1] + 0.5*diff(xb) plot(x, h); 8

9 In [10]: f = 1.* h / h.sum() plot(x,f); 9

10 In [11]: c = f.cumsum() plot(x,c); 10

11 In [12]: u = np.random.random(2000) cbins = np.concatenate(([0],c)) h2, cb = np.histogram(u, bins=cbins) plot(x, h, label="measurement data"); plot(x, h2,label="random samples from measurement data"); ylim(0, h.max()*1.4) legend(); 11

12 500 Measurement data Random samples from measurement data In [ ]: 12