Controlling for mul2ple comparisons in imaging analysis. Where we re going. Where we re going 8/15/16

Transcription

1 Controlling for mul2ple comparisons in imaging analysis Wednesday, Lecture 2 Jeane?e Mumford University of Wisconsin - Madison Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output 1

2 Review of hypothesis tes2ng What is H0? What is HA? What are the steps of carrying out a hypothesis test? Review of hypothesis tes2ng What is H0? What is HA? What are the steps of carrying out a hypothesis test? Steps of hypothesis tes2ng

3 Steps of hypothesis tes2ng Steps of hypothesis tes2ng Steps of hypothesis tes2ng What do we compare this area to (p- value)?

4 What does the p- value mean? p = 0.01 What does the p- value mean? p = 0.01 If the null distribu2on is true What does the p- value mean? p = 0.01 If the null distribu2on is true The probability of observing my sta2s2c (or something more extreme than it) is

5 What does the p- value threshold imply? We choose 0.05 Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis What does the p- value threshold imply? We choose 0.05 Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis What does the p- value threshold imply? We choose 0.05 Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis 5

6 Type I error Assuming the null is true, the probability that we reject the null Type I error Assuming the null is true, the probability that we reject the null 5% of the 2me, we ll have a false posi2ve Interpreta2on 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total 6

7 Interpreta2on 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 5% type I error - > 50 false posi2ves What we know (test results) Non- ac2ve Ac2ve Total Declared ac2ve Declared inac2ve Total Interpreta2on 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 5% type I error - > 50 false posi2ves What we don t know (truth) Non- ac2ve Ac2ve Total Declared ac2ve Declared inac2ve Total Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve 1000 Ac2ve 100 Total

8 Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve 1000 Ac2ve Total 1100 Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve 1000 Ac2ve 80 (Power) 20 (Type II err.) 100 Total 1100 Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve 80 (Power) 20 (Type II err.) 100 Total

9 Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve 50 (Type I err.) 950 (Correct) 1000 Ac2ve 80 (Power) 20 (Type II err.) 100 Total 1100 Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi2ves Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total Interpreta2on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) focus is on 5% type I error - > 50 false posi2ves controlling this number Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total

10 Implica2on of type I error If you run enough tests, you ll find something that is significant This doesn t mean it is truly significant If you run 20 tests with a 5% threshold on type I errors, you expect to have at least 1 significant test This would be a false posi2ve Hypothesis Tes2ng in fmri Mass Univariate Modeling Fit a separate model for each voxel Look at images of sta2s2cs Apply Threshold Assessing Sta2s2c Images What threshold will show us signal? High Threshold Med. Threshold Low Threshold t > 5.5 t > 3.5 t > 0.5 Good Specificity Poor Power (risk of false negatives) Poor Specificity (risk of false positives) Good Power 10

11 Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output Voxel level Levels of inference Cluster level Peak level Set level Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa2al specificity The null hyp. at a single voxel can be rejected Statistic values space 11

12 Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa2al specificity The null hyp. at a single voxel can be rejected u α space Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa2al specificity The null hyp. at a single voxel can be rejected u α space Significant Voxels No significant Voxels Cluster- level Inference Two step- process Define clusters by arbitrary threshold u clus u clus space 12

13 Cluster- level Inference Two step- process Define clusters by arbitrary threshold u clus Retain clusters larger than α- level threshold k α u clus space Cluster not significant k α k α Cluster significant Cluster- level Inference Typically be?er sensi2vity Worse spa2al specificity The null hyp. of en2re cluster is rejected Only means that one or more of voxels in cluster ac2ve u clus space Cluster not significant k α k α Cluster significant Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u clus space 13

14 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u clus Z 1 Z 2 Z 3 Z 5 space Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space 14

15 Set level inference Is there any ac2va2on anywhere in the brain? Omnibus hypothesis test of all voxels, simultaneously If significant, we only know there s ac2va2on somewhere in the brain Voxel level Levels of inference Cluster level Peak level Set level Ques2ons for you Why do some approaches require 2 thresholds? What thresholding strategy do people typically use? 15

16 Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output What error rate should we control? Per comparison error rate? Family wise error rate? False discovery rate? Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed ac2ve FWER Family wise error rate Controls the probability of any false posi2ves Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding 16

17 Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed ac2ve FWER Family wise error rate Controls the probability of any false posi2ves Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding Different types of error rates FDR False discovery rate Of the voxels you deemed significant, what percentage were null FWER vs FDR FWER P(# true null declared ac2ve > 1) FDR E (# of true null declared ac2ve / # voxels declared ac2ve) Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total

18 FWER vs FDR FWER P(# true null declared ac2ve > 1) FDR E (# of true null declared ac2ve / # voxels declared ac2ve) Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total False Discovery Rate Illustra2on: Noise Signal Signal+Noise Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives 18

19 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives Considera2ons with mul2ple comparisons What sta2s2c you re working with Voxel wise? Cluster wise? What error rate you re controlling Per comparison error rate Family wise error rate False discovery rate 19

20 Correlated data Images typically have correlated voxels # of false posi2ves = 0.05 x (# of independent tests) Extreme example Data are smoothed so much all voxels are iden2cal Only 1 out of 20 data sets would have a false posi2ve Correlated data Images typically have correlated voxels # of false posi2ves = 0.05 x (# of independent tests) Extreme example Data are smoothed so much all voxels are iden2cal Only 1 out of 20 data sets would have a false posi2ve Correlated data Coun2ng false posi2ves becomes tricky, since you don t know the number of independent things 20

21 When data are not correlated P- values computed from simulated null data When data are not correlated Thresholded p< > 4.7% are false posi2ves Correlated data Coun2ng false posi2ves becomes tricky, since you don t know the number of independent things 21

22 Same demo, with smoothed data Thresholded p- value map - > 4.2% are FP Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output FWER FWER P(# true null declared ac2ve > 1) FDR E (# of true null declared ac2ve / # voxels declared ac2ve) Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total

23 FWER Correc2on - Bonferroni Based on the Bonferroni inequality nx P (E 1 or E 2 or...e n ) apple P (E i ) i=1 If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 = FWER Correc2on - Bonferroni Based on the Bonferroni inequality nx P (E 1 or E 2 or...e n ) apple P (E i ) i=1 If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 = FWER Correc2on - Bonferroni Based on the Bonferroni inequality nx P (E 1 or E 2 or...e n ) apple P (E i ) i=1 If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

24 FWER Correc2on - Bonferroni Based on the Bonferroni inequality nx P (E 1 or E 2 or...e n ) apple P (E i ) i=1 If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 = FWER Correc2on - Bonferroni Can be too conserva2ve Bonferroni assumes all tests are independent fmri data tend to be spa2ally correlated # of independent tests < # voxels Smooth data How will the Bonferroni correc2on work with smoothed data? Will false posi2ve rate increase or decrease? 24

25 Ques2ons Why doesn t Bonferroni work well with our imaging data? Why does smoothness make mul2ple comparison correc2on more tricky? FWER Random Field theory Parametric approach to controlling false posi2ves Parametric = there s an equa2on that will spit out the p- value Beyond the scope of this course Tends to be as conserva2ve as Bonferroni FWER Random Field theory Parametric approach to controlling false posi2ves Parametric = there s an equa2on that will spit out the p- value Beyond the scope of this course Tends to be as conserva2ve as Bonferroni 25

26 FWER Random Field theory Parametric approach to controlling false posi2ves Parametric = there s an equa2on that will spit out the p- value Voxelwise version tends to be as conserva2ve as Bonferroni FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α 26

27 FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α 27

28 FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α FWER with max sta2s2c FWER & distribu2on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α 28

29 FWER MTP Solu2ons: Random Field Theory Euler Characteris2c χ u Topological Measure #blobs - #holes At high thresholds, just counts blobs No holes Never more than 1 blob Random Field FWER = P(Max voxel u H o ) = P(One or more blobs H o ) P(χ u 1 H o ) E(χ u H o ) Threshold Suprathreshold Sets Distribution details Math is hairy! Nichols and Hayasaka 2003 Cao and Worsley 2001 What you need to know Depends on smoothness of your image Must quantify smoothness and it is important to report when using RFT General idea E(χ u ) Mathy stuff *Volume/Smoothness We know what the volume is What is smoothness? 29

30 Smoothness How smooth are the data? Measured by FWHM=[FWHM x, FWHM y, FWHM z ] Starting with white noise smooth with a gaussian How large does the variance of that gaussian need to be such that the smoothness matches your data? RESEL RESolution Element RESEL=FWHM x x FWHM y x FWHM z RESEL count If your voxels were the size of a RESEL, how many are required to fill your volume? 10 voxels, 2.5 voxel FWHM smoothness 4 RESELS voxels FWHM= 2.5 voxels RESEL count=4 30

31 Note about RESEL count Not the number of independent tests Not the magic bullet for a better Bonferroni Re-expression of volume in terms of smoothness We need it, since it is necessary to calculate our p-values Revisit distribution E(χ u ) Mathy stuff *Volume/Smoothness Smoothness is defined in RESELs E(χ u ) is our p-value How does a p-value change as volume increases? How does a p-value change as smoothness increases? RFT adapts For larger volumes it is more strict Multiple comparison problem is worse For smoother data it is less strict Multiple comparison problem is less severe 31

32 Shortcomings of RFT Requires estimating a lot of parameters Random field must be sufficiently smooth If you don t spatially smooth the data enough, RFT doesn t work well I ll cover the Eklund paper later on today! Bonferroni and RFT u RF = 9.87 u Bonf = sig. vox. t 11 Sta2s2c, RF & Bonf. Threshold RFT Voxelwise RFT is rarely used in prac2ce Too conserva2ve Cluster wise RFT is very common We ll learn about cluster stats with permuta2on tes2ng 32

33 FYI If you re using RFT, you probably shouldn t lower the cluster forming threshold Assump2ons could break down If you really want to lower it, switch to nonparameteric approaches SnPM Randomise Ques2ons for you Why do we use the max sta2s2c for mul2ple comparison correc2on? Was this a voxelwise or clusterwise approach? Parametric vs Nonparametric Parametric Assume distribu2on shape Typically 1 or more parameters must be es2mated Nonparametric No assump2on on distribu2on shape Use data to construct distribu2on Related to bootstrap and jackknife, BUT not the same!!! 33

34 Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output Permuta2on test Generally can be used when the true distribu2on shape is unknown Data don t follow a normal distribu2on Generally doesn t control for mul2ple comparisons Using in conjunc2on with the max sta2s2c tackles 2 problems Not knowing the structure of the distribu2on Control FWER Permuta2on test Generally can be used when the true distribu2on shape is unknown Data don t follow a normal distribu2on Generally doesn t control for mul2ple comparisons Using in conjunc2on with the max sta2s2c tackles 2 problems Not knowing the structure of the distribu2on Control FWER 34

35 Permuta2on test Generally can be used when the true distribu2on shape is unknown Data don t follow a normal distribu2on Generally doesn t control for mul2ple comparisons Using in conjunc2on with the max sta2s2c tackles 2 problems Not knowing the structure of the distribu2on Control FWER Permuta2on test Without using max sta2s2c So we understand how it generally works With max sta2s2c So we understand how to control FWER Permuta2on test Parametric methods Assume distribu2on of sta2s2c under null hypothesis Nonparametric methods Use data to find distribu2on of sta2s2c under null hypothesis Any sta2s2c! 5% Parametric Null Distribu2on 5% Nonparametric Null Distribu2on 35

36 Permuta2on Test Toy Example Data from voxel in visual s2m. experiment A: Ac2ve, flashing checkerboard B: Baseline, fixa2on 6 blocks, ABABAB Just consider block averages... A B A B A B Null hypothesis H o No experimental effect, A & B labels arbitrary Sta2s2c Mean difference Permuta2on Test Toy Example Under H o Consider all equivalent relabelings AAABBB ABABAB BAAABB BABBAA AABABB ABABBA BAABAB BBAAAB AABBAB ABBAAB BAABBA BBAABA AABBBA ABBABA BABAAB BBABAA ABAABB ABBBAA BABABA BBBAAA Permuta2on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta2s2c values AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA

37 Permuta2on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta2s2c values Find 95%ile of permuta2on distribu2on AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA Permuta2on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta2s2c values Find 95%ile of permuta2on distribu2on AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA Permuta2on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta2s2c values Find 95%ile of permuta2on distribu2on

38 Small Sample Sizes Permutation test doesn t work well with small sample sizes Possible p-values for previous example: 0.05, 0.1, 0.15, 0.2, etc Tends to be conservative for small sample sizes Permuta2on Test & Exchangeability Exchangeability is fundamental Def: Distribu2on of the data unperturbed by permuta2on Under H 0, exchangeability jus2fies permu2ng data Allows us to build permuta2on distribu2on Permuta2on Test & Exchangeability Subjects are exchangeable Under Ho, each subject s A/B labels can be flipped fmri scans are not exchangeable under Ho If no signal, can we permute over 2me? No, permu2ng disrupts order, temporal autocorrela2on 38

39 Permuta2on Test & Exchangeability Two sample t test Compare subjects in group 1 to subjects in group 2 Randomly assign group labels in permuta2ons One sample t test Randomly flip sign of values for some subjects Ques2ons for you What is permuted for a 1- sample t- test? What is permuted for a 2- sample t- test? What is permuted for a correla2on? Why are small sample sizes problema2c for permuta2on tes2ng? Controlling FWER: Permuta2on Test Parametric methods Assume distribu2on of max sta2s2c under null hypothesis Nonparametric methods Use data to find distribu2on of max sta2s2c under null hypothesis Again, any max sta2s2c! 5% Parametric Null Max Distribu2on 5% Nonparametric Null Max Distribu2on 39

40 Permuta2on Test Other Sta2s2cs Collect max distribu2on To find threshold that controls FWER Consider smoothed variance t sta2s2c To regularize low- df variance es2mate Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 40

41 Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 41

42 Max sta2s2c for imaging data 1. Compute your sta2s2cs map for original data 2. Shuffle labels and compute sta2s2cs map 3. Save the largest sta2s2c over the whole brain 4. Repeat steps 2-3 many 2mes ( ,000) 5. Use distribu2on of stats over permuta2ons to compute threshold 6. Apply threshold to map from step 1 Permuta2on Test Smoothed Variance t Collect max distribu2on To find threshold that controls FWER Consider smoothed variance t sta2s2c mean difference variance t-statistic Permuta2on Test Smoothed Variance t Collect max distribu2on To find threshold that controls FWER Consider smoothed variance t sta2s2c mean difference smoothed variance Smoothed Variance t-statistic 42

43 Permuta2on Test Example fmri Study of Working Memory 12 subjects, block design Marshuetz et al (2000) Item Recogni2on Ac2ve: View five le?ers, 2s pause, view probe le?er, respond Baseline: View XXXXX, 2s pause, view Y or N, respond Second Level RFX Difference image, A- B constructed for each subject One sample t test UBKDA Active... Baseline XXXXX D yes... N no Permuta2on Test Example Permute! 2 12 = 4,096 ways to flip 12 A/B labels For each, note maximum of t image. Permuta2on Distribu2on Maximum t Maximum Intensity Projec2on Thresholded t u Perm = sig. vox. t 11 Sta2s2c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta2s2c, RF & Bonf. Threshold 378 sig. vox. Smoothed Variance t Sta2s2c, Nonparametric Threshold RFT threshold is conservative (not smooth enough, d.f. too small) Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f. 43

44 u Perm = sig. vox. t 11 Sta2s2c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta2s2c, RF & Bonf. Threshold 378 sig. vox. Smoothed Variance t Sta2s2c, Nonparametric Threshold RFT threshold is conservative (not smooth enough, d.f. too small) Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f. u Perm = sig. vox. t 11 Sta2s2c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta2s2c, RF & Bonf. Threshold 378 sig. vox. Smoothed Variance t Sta2s2c, Nonparametric Threshold RFT threshold is conservative (not smooth enough, d.f. too small) Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f. u Perm = sig. vox. t 11 Sta2s2c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta2s2c, RF & Bonf. Threshold 378 sig. vox. Smoothed Variance t Sta2s2c, Nonparametric Threshold RFT threshold is conservative (not smooth enough, d.f. too small) Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f. 44

45 Permuta2on test cluster sta2s2c Two step- process Define clusters by arbitrary threshold u clus u clus space Permuta2on test cluster sta2s2c Two step- process Define clusters by arbitrary threshold u clus Retain clusters larger than α- level threshold k α u clus space Cluster not significant k α k α Cluster significant Permuta2on test cluster sta2s2cs Cluster size Simply count how many voxels are in the sta2s2c Cluster mass Sum up the sta2s2cs in the cluster 45

46 Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold 46

47 Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold 47

48 Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold Permuta2on test cluster sta2s2cs 1. Find clusters with original data 2. Permute labels 3. Compute sta2s2cs 4. Apply cluster- forming threshold 5. Compute cluster sta2s2cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many 2mes ( ,000) 8. Use distribu2on from step 7 to find cluster (size or mass) threshold & apply to step 1 Ques2ons for you Why don t permuta2on tests, alone, fix mul2ple comparisons? What did we need to use to address mul2ple comparisons? How are the voxelwise and clusterwise permuta2on tests set up? 48

49 Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output FWER vs FDR FWER P(# true null declared ac2ve > 1) FDR E (# of true null declared ac2ve / # voxels declared ac2ve) Declared ac2ve Declared inac2ve Total Non- ac2ve Ac2ve Total Controlling FDR Tends to be less conserva2ve than controlling FWER What rate is appropriate? Imagers use 5%...out of habit FDR people I ve met outside of imaging ozen use higher values Decide before you threshold your data Don t choose what makes your data look good 49

50 /15/16 Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 i/v 1 Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α Reject all hypotheses corresponding to p (1),..., p (r). p-value 0 p (i) i/v α 1 i/v Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α Reject all hypotheses corresponding to p (1),..., p (r). p-value 0 p (i) i/v α i/v 1 50

51 FDR Example FWER Perm. Thresh. = voxels FDR Threshold = ,073 voxels Where we re going Review of hypothesis tes2ng introduce mul2ple tes2ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela2ng all of this to SPM output Guess what? Now you have the knowledge needed to understand a huge/daun2ng table SPM spits out! Let s do it 51

52 SPM output SPM output Which level of inference is missing? SPM output what exci2ng conclusion can we make? 52

53 SPM output Recall: FWE correc2on shown earlier was super conserva2ve compared to FDR. Why does this look different? SPM output What do you think K E is? What sta2s2c does the p- value correspond to? SPM output The uncorrected stat doesn t take the search volume into account 53

54 SPM output See the note at the bo?om? SPM output Do any clusters have more than one peak? SPM output Last, but not least, you ll use this in lab. This is used to threshold clusters so you can look at only the significant ones 54

55 SPM output Compare this threshold to the FWE p- values for cluster stats That s it! Ques2ons? 55