Linear Modeling with Bayesian Statistics

Transcription

1 Linear Modeling with Bayesian Statistics Bayesian Approach I I I I I Estimate probability of a parameter State degree of believe in specific parameter values Evaluate probability of hypothesis given the data Incorporate prior knowledge Recognizes that Data is one realization of some parameter distribution

2 Today Markov Chain Monte-Carlo Sampling Applying MCMC & Bayesian Inference to Multiparameter Models Evaluating convergence and fit of MCMC fit models Modifying our Prior Bayes Theorem Expanded p(θ X) = p(θ X) = p(x θ)p (θ) j i=0 p(x θ i )p(θ i ) - Algebraically Solvable p(x θ)p (θ) p(x θ)p(θ)dθ - Analytically Solveable for Conjugate Priors p(x θ)p (θ η)p(η)dη p(θ X) = p(x θ)p(θ)dθdη - Hierarchical Model: need numerical integration approach with random hyperparameters

3 Trace of (Intercept) Density of (Intercept) Iterations N = 1000 Bandwidth = If we cannot analytically solve a distribution, we can still simulate from it: Chose a set of starting values X at t=0 Chose a random set of parameters, Y, based on X Calculate an acceptance ratio, α, based on P(Y)/P(X) If α 1 X(t+1) = Y Otherwise, select a uniorm random number between 0 and 1, U If U α, X(t+1) = Y. Otherwise, X(t+1) = X. Rinse and repeat (Note, this is the Metropolis-Hastings Algorithm - there are others)

4 This is a time series. To use it for inference to sample from the final stationary distribution: Discard a burn in set of samples Thin your chain to reduce temporal autocorrelation Examine chain for convergence on your posterior distribution Evaluate multiple chains to ensure convergence to a single distribution Many different samplers using different decision rules for f. We use the Gibbs Sampler commonly. Value Iteration

5 Value Iteration Value Iteration

6 Value Iteration Value Iteration

7 Value Iteration Value Iteration

8 Thinned Value Iteration Linear Modeling with MCMC

9 Software Options for MCMC WinBUGS OpenBUGS JAGS STAN MCMCglmm in R MCMCpack in R BUGS code for a Simple Linear Regression model { Prior alpha dnorm(0,0.001) beta dnorm(0,0.001) sigma dunif(0,100) Likelihood for (i in 1:n){ y[i] dnorm(mu[i],tau) mu[i] <- alpha + beta*x[i] } }

10 Back to the Wolves with MCMCglmm & coda pups inbreeding.coefficient MCMCglmm Powerful package that fits bayesian models using Metropolis-Hastings MCMC Available tools to tweak priors Flexible in error distribution Can accomodate random effects Built to deal with phylogenetic autocorrelation Can handle other autocorrelation (temporal, spatial) as well

11 MCMCglmm v. LM wolf_lm <- lm(pups inbreeding.coefficient, data=wolves) wolf_mcmc <- MCMCglmm(pups inbreeding.coefficient, data=wolves, verbose=false) Coefficients are Very Similar summary(wolf_mcmc)$solutions post.mean l-95% CI u-95% CI eff.samp (Intercept) inbreeding.coefficient pmcmc (Intercept) inbreeding.coefficient summary(wolf_lm)$coef Estimate Std. Error t value (Intercept) inbreeding.coefficient Pr(> t ) (Intercept) 3.143e-08 inbreeding.coefficient 1.633e-03

12 All of the assumptions of a linear model with a normal error distribution still apply! Make a QQ-Plot! wolf_resid <- wolves$pups - predict(wolf_mcmc) qqnorm(wolf_resid) qqline(wolf_resid) Normal Q Q Plot Sample Quantiles

13 Check Normality of Residuals Chisq - problems with small values q <- qqnorm(wolf_resid) Normal Q Q Plot Sample Quantiles Theoretical Quantiles chisq.test(q$y, q$x) Fitted v. Residuals! Pearson's Chi-squared test plot(fitted(wolf_mcmc), wolf_resid) abline(a=0, b=0, lwd=1, lty=2) data: q$y and q$x X-squared = 432, df = 414, p-value = KolmogorovâĂŞSmirnov Problems with Ties ks.test(wolf_resid) Error: argument "y" is missing, with no default wolf_resid Shapiro-Wilks Test Very Sensitive shapiro.test(wolf_resid) Shapiro-Wilk normality test data: wolf_resid fitted(wolf_mcmc)

14 MCMC has its own assumptions! No Inferences Without Convergence!!! coda package Analysis of MCMC chains Variety of tools for convergence Works with single or multiple chains

15 Does this look Converged? plot(wolf_mcmc, trace=true, smooth=t) Trace of (Intercept) Density of (Intercept) Iterations N = 1000 Bandwidth = Trace of inbreeding.coefficient Density of inbreeding.coefficient Iterations N = 1000 Bandwidth = Trace of units Density of units Multiple Chains Every MCMC run (chain) starts with different start values Each run SHOULD converge on the same answer Chains should overlap make a list of MCMC chains using Iterations Sol which contains parameters N = 1000 Bandwidth = wolfchains2 <- as.mcmc.list(lapply(1:2, function(i) MCMCglmm(pups inbreeding.coefficient, data=wolves, verbose=false)$sol ))

16 Multiple Chains plot(wolfchains2) Trace of (Intercept) Density of (Intercept) Iterations N = 1000 Bandwidth = Trace of inbreeding.coefficient Density of inbreeding.coefficient Iterations N = 1000 Bandwidth = Multiple Chains Trace of (Intercept) Density of (Intercept) Iterations N = 1000 Bandwidth = Trace of inbreeding.coefficient Density of inbreeding.coefficient Iterations N = 1000 Bandwidth = 0.766

17 New Information from Many Chains Iterations = 3001:12991 Thinning interval = 10 Number of chains = 2 Sample size per chain = Empirical mean and standard deviation for each variable, plus standard error of the mean: Mean SD Naive SE Time-series SE (Intercept) inbreeding.coefficient Quantiles for each variable: 2.5% 25% 50% 75% 97.5% (Intercept) inbreeding.coefficient But am I REALLY Converging? Gelman-Rubin Diagnostic Averaging across chains should shrink estimates to a mean How much shrinkage is happening after 10, 20, 30, 40,... iterations? If you have converged, the scale reduction should be 1 - i.e., none. Other diagnostics, too - but this is fairly standard

18 Gelman-Rubin Plots gelman.plot(wolfchains4) (Intercept) inbreeding.coefficient shrink factor median 97.5% shrink factor median 97.5% last iteration in chain last iteration in chain Gelman-Rubin By the Numbers gelman.diag(wolfchains4) Potential scale reduction factors: Point est. Upper C.I. (Intercept) inbreeding.coefficient Multivariate psrf 1

19 So what about that Prior thing? Priors The Default Prior for an Intercept-Slope Model: MV Norm ( µ = ( ) 0, V = 0 Intercept = row 1, Slope = Row2 ( ) 1e10 0 ) 0 1e10

20 What if we have a prior? MV Norm ( µ = ( ) 0, V = 10 ( ) 1e10 0 ) 0 1 strongprior<-list(b=list(mu=c(0,-5), V=diag(c(1e10, 1)))) Put a Prior On It wolf_mcmc_prior <- MCMCglmm(pups inbreeding.coefficient, data=wolves, verbose=false, prior=strongprior)

21 Influence of a Prior on this Small Dataset summary(wolf_mcmc)$solutions post.mean l-95% CI u-95% CI eff.samp (Intercept) inbreeding.coefficient pmcmc (Intercept) inbreeding.coefficient summary(wolf_mcmc_prior)$solutions post.mean l-95% CI u-95% CI eff.samp (Intercept) inbreeding.coefficient pmcmc (Intercept) inbreeding.coefficient 0.001