Statistical Methods in functional MRI. Standard Analysis. Data Processing Pipeline. Multiple Comparisons Problem. Multiple Comparisons Problem

Transcription

1 Statistical Methods in fnctional MRI Lectre 7: Mltiple Comparisons 04/3/13 Martin Lindqist Department of Biostatistics Johns Hopkins University Data Processing Pipeline Standard Analysis Data Acqisition Reconstrction Experimental Design Preprocessing Slice-time Correction Data Analysis Localizing Brain Activity 1. Fit a statistical model (e.g., the GLM) to a certain voxel in the brain.. Use the estimated model parameters to test for an effect of interest, i.e. H 0 : 1- =0. 3. Repeat steps 1 and for each voxel. 4. Smmarize the reslts in a statistical image. Motion Correction, Co-registration & Normalization Spatial Smoothing Connectivity Prediction 5. Determine which voxels show a statistically significant effect, i.e. threshold the image. Mltiple Comparisons Problem Choosing an appropriate threshold is complicated by the fact we are dealing with a family of tests. If more than one hypothesis test is performed, the risk of making at least one Type I error is greater than the vale for a single test. Mltiple Comparisons Problem Which of 100,000 voxels are significant? =0.05 5,000 false positive voxels Choosing a threshold is a balance between sensitivity (tre positive rate) and specificity (tre negative rate). t > 0.5 t > 1.5 t >.5 t > 3.5 t > 4.5 t > 5.5 t > 6.5 t > 0.5 t > 1.5 t >.5 t > 3.5 t > 4.5 t > 5.5 t > 6.5 The more tests one performs, the greater the likelihood of getting at least one false positive. 1

2 Measres of False Positives There exist several ways of qantifying the likelihood of obtaining false positives. Family-Wise Error Rate (FWER) Probability of any false positives False Discovery Rate (FDR) Proportion of false positives among rejected tests Family-Wise Error Rate The family-wise error rate (FWER) is the probability of making one or more Type I errors in a family of tests, nder the nll hypothesis. FWER controlling methods: Bonferroni correction Random Field Theory Permtation Tests Problem Formlation Let H 0i be the hypothesis that there is no activation in voxel i, where i V ={1,.m}. Let T i be the vale of the test statistic at voxel i. The family-wise nll hypothesis, H 0, states that there is no activation in any of the m voxels. H 0 H 0i i V If we reject a single voxel nll hypothesis, H 0i, we will reject the family-wise nll hypothesis. A false positive at any voxel gives a Family-Wise Error (FWE) Assming H 0 is tre, we want the probability of falsely rejecting H 0 to be controlled by, i.e. Bonferroni Correction Choose the threshold so that Example Generate voxels from an iid N(0,1) distribtion Hence, P T i H 0 m Threshold at =1.645 i P Ti H0 Boole s Ineqality i m Approximately 500 false positives.

3 Bonferroni Correction To control for a FWE of 0.05, the Bonferroni correction is 0.05/10,000. The Bonferroni correction is very conservative, i.e. it reslts in very strict significance levels. This corresponds to =4.4. On average only 5 ot of every 100 generated in this fashion will have one or more vales above. It decreases the power of the test (probability of correctly rejecting a false nll hypothesis) and greatly increases the chance of false negatives. It is not optimal for correlated data, and most fmri data has significant spatial correlation. No false positives Spatial Correlation We may be able to choose a more appropriate threshold by sing information abot the spatial correlation in the data. Random field theory allows one to incorporate the correlation into the calclation of the appropriate threshold. Maximm Statistic Link between FWER and max statistic. FWER = P(FWE) = P( i {T i } H o ) P(any t-vale exceeds nder nll) = P( max i T i H o ) P(max t-vale exceeds nder nll) It is based on approximating the distribtion of the maximm statistic over the whole image. Choose the threshold sch that the max only exceeds it % of the time Random Field Theory A random field is a set of random variables defined at every point in D-dimensional space. A Gassian random field has a Gassian distribtion at every point and every collection of points. A Gassian random field is defined by its mean fnction and covariance fnction. Random Field Theory Consider a statistical image to be a lattice representation of a continos random field. Random field methods are able to: approximate the pper tail of the maximm distribtion, which is the part needed to find appropriate thresholds, and accont for the spatial dependence in the data. 3

4 Random Field Theory Eler Characteristic Consider a random field Z(s) defined on s D R where D is the dimension of the process. Eler Characteristic A property of an image after it has been thresholded. Conts #blobs - #holes At high thresholds, jst conts #blobs = =.75 1 = 3.5 Random Field Threshold Controlling the FWER Link between FWER and Eler Characteristic. FWER = P(max i T i H o ) = P(One or more blobs H o ) P( 1 H o ) E( H o ) no holes exist never more than 1 blob Closed form reslts exist for E( ) for Z, t, F and continos random fields. 3D Gassian Random Fields For large search regions: where E( R R(4log) Here V is the volme of the search region and the fll width at half maximm (FWHM) represents the smoothness of the image estimated from the data. R = Resoltion Element (Resel) ) x 3/ V FWHM FWHM ( y 1) e FWHM z Controlling the FWER RFT Assmptions For large : where FWER R R(4log) V FWHM FWHM x 3/ y ( 1) e FWHM z The entire image is either mltivariate Gassian or derived from mltivariate Gassian images. The statistical image mst be sfficiently smooth to approximate a continos random field. FWHM smoothness 3-4 voxel size. Properties: - As increases, FWER decreases (Note large). - As V increases, FWER increases. - As smoothness increases, FWER decreases. The amont of smoothness is assmed known. Estimate is biased when images not sfficiently smooth. Several layers of approximations. 4

5 Levels of Inference A statistical map has many topological attribtes that can be sed to categorize it. Height of a peak Nmber of peaks The volme or extent of an excrsion set Voxel-level Inference Retain voxels above -level threshold We can perform corrections on several levels. Voxel-level Clster-level Significant Voxels space No significant Voxels Clster-level Inference Two step-process Define clsters by arbitrary threshold cls Retain clsters larger than -level threshold k Clster-level Inference Can se RFT to perform clster level inference. Assme clsters behave like a mltidimensional Poisson point process. The probability of getting one or more clsters of volme k is given by: p 1 exp{ E( ) P( n k)} Clster not significant cls k k space Clster significant Here E( ) is the expected Eler Characteristic and n represents the volme of a clster. SnPM Illstration Statistical nonparametric mapping (Nichols & Holmes) is a nonparametric eqivalent to SPM. It ses a permtation test, rather than random field theory, to correct for mltiple comparisons. It allows one to avoid the assmptions needed for sing random field theory. Data from V1 voxel in visal stimls experiment A: Active, flashing checkerboard B: Baseline, fixation 6 blocks, ABABAB Jst consider block averages. A B A B A B Nll hypothesis H 0 No experimental effect, i.e. the A and B labels are arbitrary. Statistic Mean difference between conditions. Nichols & Holmes 5

6 Under H o Consider all eqivalent re-labelings. Assme exchangeability, i.e. the distribtion of the statistic is the same whatever the relabeling. Compte all possible statistic vales. Determine the permtation distribtion. Each relabeling is eqally likely. Hence, each statistic has eqal probability. Permtation distribtion P-vale = 0.05 AAABBB 4.8 ABABAB 9.45 BAAABB BABBAA AABABB -3.5 ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.5 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA Actal vale = 9.45 Permtation test don t work very well with small sample sizes, as they tend to be conservative. Comments Reqires only the assmption of exchangeability Under H 0, the distribtion is nchanged by permtation. This allows s to bild the permtation distribtion. Sbjects are exchangeable Under H 0, each sbject s A/B labels can be flipped. Permtation tests sefl for nd level analysis. Isses Sample size If there are N possible relabelings, the smallest attainable p- vale is 1/N which can be problematic at small N. If there are too many possible relabelings it may not be feasible to compte the statistic images for all of them. Randomly sample from the poplation of relabelings. Each relabeling shold be eqally likely to be chosen. fmri scans not exchangeable nder H 0 The problem is temporal atocorrelation. Be carefl performing permtation tests on individal sbjects. Flexibility The permtation approach is free to consider any statistic and is not bond to those that have a known distribtional form. Isses Comptational Intensity Analysis needs to be repeated for each relabeling Implementation Each experimental design type needs niqe code to generate permtations Power and Validity Nonparametric procedres are less powerfl than their parametric conterparts when the assmptions of the latter hold. If assmptions not met, nonparametric procedres more valid. Mltiple Comparisons Permtation tests can be sed to control for mltiple comparisons. To do so all voxels need to be considered simltaneosly. Argments can be extended to image-level inference by considering an appropriate max statistic. Permtations carried ot on the image level, i.e. entire images are relabeled. 6

7 Illstration A B A B A B Controlling FWER A single threshold test thresholds the statistic image at a critical vale and voxels exceeding the threshold are deemed active. Max statistic A-B By repeatedly permting the image labels and compting the max statistic we obtain the permtation distribtion needed for correcting the FWER. Procedre: Re-label the images. Compte the statistic for each voxel. Compte the maximm statistic over the whole image. Repeat to constrct the permtation distribtion of the max statistic. Controlling FWER A sprathreshold clster test thresholds the statistic image at a primary threshold, and then stdies the extent of activation in a second level. Procedre: Re-label the images. Compte the statistic for each voxel. Threshold at a primary threshold. Compte the size of the largest clster above the threshold. Repeat to constrct the permtation distribtion. d = d = 1 Power Bonferroni and SPM s GRF correction do not accont accrately for spatial smoothness with the sample sizes and smoothness vales typical in imaging experiments. Nonparametric tests more accrate and more sensitive. d = 0.5 d = d = 1 d = 0.5 Wager et al. Isses with FWER Methods that control the FWER (Bonferroni, RFT, Permtation Tests) provide a strong control over the nmber of false positives. While this is appealing the reslting thresholds often lead to tests that sffer from low power. Power is critical in fmri applications becase the most interesting effects are sally at the edge of detection. False Discovery Rate The false discovery rate (FDR) is a recent development in mltiple comparison problems de to Benjamini and Hochberg (1995). While the FWER controls the probability of any false positives, the FDR controls the proportion of false positives among all rejected tests. 7

8 p-vale 0 1 Notation Sppose we perform tests on m voxels. Declared Inactive Declared Active Trly inactive U V m 0 Trly active T S m-m 0 m-r R m In this notation: FWER False discovery rate: FDR Definitions P V V E R 1 U, V, T and S are nobservable random variables. The FDR is defined to be 0 if R=0. R is an observable random variable. Properties A procedre controlling the FDR ensres that on average the FDR is no bigger than a prespecified rate q which lies between 0 and 1. However, for any given data set the FDR need not be below the bond. BH Procedre 1. Select desired limit q on FDR (e.g., 0.05). Rank p-vales, p (1) p ()... p (m) 3. Let r be largest i sch that p (i) i/m q p (i) An FDR-controlling techniqe garantee controls of the FDR in the sense that FDR q. 4. Reject all hypotheses corresponding to p (1),..., p (r). i/m 0 1 q Comments The BH procedre is adaptive in the sense that the larger the signal, the lower the threshold. If all nll hypothesis are tre, the FDR is eqivalent to the FWER. Low signal High signal Any procedre that controls the FWER also controls the FDR. A procedre that controls the FDR only can be less stringent and lead to a gain in power. q m q Since FDR controlling procedres work only on the p-vales and not on the actal test statistics, it can be applied to any valid statistical test. 8

9 Example =0.10, No correction Signal FWER control at 10% Percentage of false positives + Noise = FDR control at 10% Occrrence of false positive FWER Signal + Noise Percentage of active voxels that are false positives Uncorrected Thresholds Most pblished PET and fmri stdies se arbitrary ncorrected thresholds (e.g., p<0.001). A likely reason is that with the available sample sizes, corrected thresholds are so stringent that power is extremely low. Using ncorrected thresholds is problematic when interpreting conclsions from individal stdies, as many of the activated regions may be false positives. Nll findings are hard to disseminate, hence it is difficlt to refte false positives established in the literatre. Extent Threshold Sometimes an arbitrary extent threshold is sed when reporting reslts. Here a voxel is only deemed trly active if it belongs to a clster of k contigos active voxels (e.g., p<0.001, 10 contingent voxels). Unfortnately, this does not necessarily correct the problem becase imaging data are spatially smooth and therefore false positives may appear in clsters. Example Activation maps with spatially correlated noise thresholded at three different significance levels. De to the smoothness, the false-positive activation are contigos regions of mltiple voxels. =0.10 =0.01 =0.001 Example Similar activation maps sing nll data. =0.10 =0.01 =0.001 Note: All images smoothed with FWHM=1mm Note: All images smoothed with FWHM=1mm 9

10 10