Monte Carlo Methods and Statistical Computing: My Personal E

Transcription

1 Monte Carlo Methods and Statistical Computing: My Personal Experience Department of Mathematics & Statistics Indian Institute of Technology Kanpur November 29, 2014

2 Outline Preface 1 Preface

3 Outline Preface 1 Preface

4 Limitations: Preface 1. I must admit that the topics I am going to cover are definitely not exhaustive. 2. Topics are purely of my own interest which have developed over the last 30 years. 3. I am not going to describe any statistical package.

5 Advantages: Preface 1. Packages have their own problems. 2. Different packages can give different answers even on a relatively simple problem. 3. We should know the limitations of the packages. 4. I will try to provide a general approach

6 Journals: Preface 1. Journal of Computational and Graphical Statistics. 2. Computational Statistics and Data Analysis. 3. Journal of and Simulation. 4. Statistical Computing 5. Communications in Statistics - Simulation and Computation

7 Books: Preface 1. Simulation by Sheldon Ross 2. Nonuniform Random Deviate Generator, L. Devroye 3. Simulation modelling and analysis, Law and Kelton Journal of and Simulation. 4. Statistical Computing: J.F. Keneddy and R. Gentle 5. Statistical Computing, D. Kundu and A. Basu

8 Outline Preface 1 Preface

9 Monte Carlo Method: Definition 1. A broad class of numerical alogrithm depends on repeated random sampling. 2. If it is not possible to obtain the exact analytical solution often Monta Carlo method can be used to provide a very good approximate solution

10 Monte Carlo Method: A brief history 1. It was invented by Stanislaw Ulam, a famous Polish Mathematician, in the late John von Neumann first wrote the computer code to perform Monte Carlo simulations 3. Metropolis gave this name

11 Outline Preface 1 Preface

12 Where it can be used? 1. Calculating the area below a curve. 2. Calculating multidimensional integration. 3. Optimization. 4. Analyzing any complicated stochastic system (model).

13 Examples Preface Suppose we want to compute b a e x2 dx. Or suppose we want to compute b1 bk... f(x 1,...,x k )dx 1...dx k. a 1 a k

14 Examples:Contd. Preface Suppose we want to find the maximum or minimum of the following function f(x 1,...,x k ), where a 1 x 1 b 1,...,a k x k b k. Or suppose we want to analyze the following non-linear model y(x 1,...,x k ) = f(x 1,...,x k,θ)+e.

15 Outline Preface 1 Preface

16 Back Ground Preface 1. Knowledge of Basic Probability. 2. Discrete and Continuous random variables. 3. Stochastic models. 4. Generation of random numbers.

17 Knowlwedge of Basic Probability 1. Idea of a random experiment. 2. Basic idea of convergence of random variables. 3. Weak and strong law of large numbers. 4. Central limit theorem.

18 Discrete Random Variables 1. Uniform. 2. Binomial. 3. Geometric. 4. Poisson.

19 Continuous Random Variables 1. Uniform. 2. Exponential. 3. Normal. 4. Gamma. 5. Log-concave probability density function

20 Generation of Random Numbers First we need to know how to generate Uniform random numbers. This is the most basic problem. In this respect we use group theory results and machine powers.

21 Generation of Non-Uniform Random Numbers The most popular method is the inverse transformation. The following result can be used. If X is a random variable with the distribution function F(x), then F(X) follows uniform distribution. Therefore X = F 1 (U)

22 Generation of Discrete Random Numbers All the discrete distributions can be generated using inverse transformation method. Suppose P(X = a i ) = p i, for i = 1,2,... Without loss of generality we can assume a 1 < a 2 <... Draw a uniform random number say u, if k 1 i=1 p i < u < k i=1 p i, then X takes the value a k.

23 Generation of Continuous Random Numbers Many continuous random variables can be generated using inverse transformation method, for example exponential, Weibull, generalized exponential distributions etc. On the other hand several well known distribution cannot be obtained using inverse transformation method. For example normal, gamma etc.

24 Generation of Continuous Random Numbers If a continuous distribution cannot be generated using inverse transformation method, one of the most useful method is the acceptance rejection method. The idea is as follows. If we want to generate from f(x), try to find g(x), from which generation is simple so that it satisfies the following f(x) cg(x).

25 Acceptance Rejection Method: Algorithm 1. Generate Y from g(x). 2. Generate a uniform random vaiable U. 3. If U f(y)/cg(y), set X = Y, otherwise return to 1.

26 Acceptance Rejection Method: Theorem Theorem: 1. The random variable generated by this method has density function f(x) 2. The number of iterations of the algorithm that are needed is a geometric random variable with mean c,

27 Acceptance Rejection Method: Example Example 1: Suppose we want to generate from f(x) = 20x(1 x) 3 ; 0 < x < 1. Take g(x) = 1, 0 < x < 1. c = 135/64.

28 Acceptance Rejection Method: Example Example 2: Suppose we want to generate from f(x) = 2 π x 1/2 e x ; 0 < x <. Take and g(x) = 2 3 e 2x/3 0 < x <. c = 33/2 (2πe) 1/2.

29 Acceptance Rejection Method: Example Example 3: Suppose we want to generate from f(x) = 2 2π e x2 /2 ; 0 < x <. Take and g(x) = e x ; 0 < x <. c = 2e/π.

30 Outline Preface 1 Preface

31 Very Simple Example Consider the following simple linear regression model Y = Xb+e We know the LSE s can be obtained as b = (X T X) 1 X T Y. We have a complete very nice theory when all the components of the errors are i.i.d. normal random variables.

32 Very Simple Example: Contd. Consider some slightly different conditions of the same model. 1. What will happen if the errors are not normal? 2. What will happen if the errors are heavy tail? 3. What will happen if there are outliers? 4. What will happen if the errors are correlated?

33 Very Simple Example: Contd. In all these cases Monte Carlo Method can be used to asses the performances of the estmimators. It is very simple also. 1. Generate e 2. Generate Y. 3. Calculate b. 4. Repeat step 1 to step 4, several times.

34 Example Preface Consider the following simple linear regression model Y = Xb+e Suppose we want to estimate b by minimizing the least absolute errors i.e. b = argmin Y Xb. Theories are quite complicated. All the results are asymptotic in nature.

35 Example Preface Consider the following non-linear regression model Y = f(x,θ)+e Here f is a known function the vector X is also known, the paramete vector θ is unknown. The problem is to estimate the parameter vector θ, based on a sample of size n.

36 Example Preface Natural estimators will be n θ = argmin Y i f(x i,θ) 2. or θ = argmin i=1 n Y i f(x i,θ). i=1 Theories are quite complicated. All the results are asymptotic in nature.

37 Example Preface Monte Carlo method can be used to asses the perofrmance of the estimators. Based on the Monte Carlo method the biases and the mean squared errors can be calculated. Based on Bootstrap method confidence intervals also can be obtained.

38 Example: Importance Sampling In Bayesian analysis often we need to compute the posterior mean as follows: θ = E(h(X)) = h(x)f(x)dx. Here f(x) is the PDF of X, and x can be a very high dimensional. In Bayesian analysis f(x) is the posterior density function.

39 Example: Importance Sampling Monte Carlo simulation technique can be used to approximate the value of θ as follows: θ = 1 N N h(x i ), i=1 here X 1,...,X N is a random sample of size N from f(x).

40 Example: Importance Sampling Often it is observed that it is not very easy to generate samples from f(x). h(x)f(x) θ = h(x)f(x)dx = g(x)d(x). g(x) θ = 1 N N i=1 h(x i )f(x i ), g(x i ) here X 1,...,X N is a random sample of size N from g(x).

41 Outline Preface 1 Preface

42 Important Issues Preface 1. Finding Maximum likelihood estimators in a general problem. 2. Finding least squares estimators of linear regresssion model when the design matrix is close to a singular matrix 3. Non-linear regression model if the number of parameters are very high

43 MLE Preface It basically invloves maximizing a function of the form: f(θ 1,...,θ p ) Standard method is to use Newton-Raphson method:

44 Newton-Raphson Method Assuming sufficiently smooth f(θ), we want to solve f(θ) θ = 0 Standard method is to use Newton-Raphson method. Using Taylor series expansion, it can be easily obtained: [ θ (k+1) = θ (k) 2 f(θ (k) ] 1 ) f(θ (k) ) θ θ T θ

45 Profile Likelihood Method 1. For fixed θ 1,...,θ k, try to maximize with respect to θ k+1,...,θ p 2. Maximize with respect to θ 1,...,θ k.

46 EM Algorithm Preface Suppose the data are coming from a mixture model, and we compute the MLEs of the unknown parameters f(x) = π j 0, k j=1 π j = 1. k π j f j (x;θ j ), j=1

47 Mixture Model: MLE Based on a random sample x 1,...,x n, we want to compute the MLEs of the unknown parameters L(π,θ) = n i=1 k π j f j (x i ;θ j ). j=1

48 Missing Value Problem We treat this as a missing value problem 1. Assume the data are of the form (x,δ) 2. Compute E(δ Data) 3. Continue the process

49 Copula Method Preface Any multivarite distribution can be written uniquely as follows: F(x 1,...,x p ;θ) = C(F 1 (x 1 ;θ 1 ),...,F p (x p : θ p );γ) First estimate the marginal parameters, and then estimate the copula parameters

50 Non-linear regression Preface Consider the following model y(t) = p [A k cos(ω k t)+b k sin(ω k t)]+e(t) k=1 Estimate the unknown parameters

51 Outline Preface 1 Preface

52 Very important areas Bayesian comutation: mainly MCMC Classification problem Small n large p problem Non-parametric regression Functional data analysis

53 Thank You