Controlling for multiple comparisons in imaging analysis. Wednesday, Lecture 2 Jeanette Mumford University of Wisconsin - Madison

Transcription

1 Controlling for multiple comparisons in imaging analysis Wednesday, Lecture 2 Jeanette Mumford University of Wisconsin - Madison

2 Motivation Run 100 hypothesis tests on null data using p<0.05? How many significant results will I find (on average)?

3

4 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

5 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

6 Review of hypothesis testing What is H0? What is HA? What are the steps of carrying out a hypothesis test?

7 Review of hypothesis testing What is H0? What is HA? What are the steps of carrying out a hypothesis test?

8 Steps of hypothesis testing

9 Steps of hypothesis testing

10 Steps of hypothesis testing

11 Steps of hypothesis testing What do we compare this area to (p-value)?

12 What does the p-value mean? p = 0.01

13 What does the p-value mean? p = 0.01 If the null distribution is true

14 What does the p-value mean? p = 0.01 If the null distribution is true The probability of observing my statistic (or something more extreme than it) is 0.01

15 What does the p-value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

16 What does the p-value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

17 What does the p-value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

18 1100 total voxels 100 voxels have β=δ 80% power -> 80 voxels detected 1000 voxels have β=0 Interpretation 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active Active Total

19 1100 total voxels 100 voxels have β=δ 80% power -> 80 voxels detected 1000 voxels have β=0 Interpretation 5% type I error -> 50 false positives What we know (test results) Declared active Fail to Declare active Total Non-active Active Total

20 1100 total voxels 100 voxels have β=δ 80% power -> 80 voxels detected 1000 voxels have β=0 Interpretation 5% type I error -> 50 false positives What we don t know (truth) Non-active Active Total Declared active Fail to Declare active Total

21 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active 1000 Active 100 Total 1100

22 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active 1000 Active Total 1100

23 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active 1000 Active 80 (Power) 20 (Type II err.) 100 Total 1100

24 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active Active 80 (Power) 20 (Type II err.) 100 Total 1100

25 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active 50 (Type I err.) 950 (Correct) 1000 Active 80 (Power) 20 (Type II err.) 100 Total 1100

26 Interpretation 1100 total voxels 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives Declared active Fail to Declare active Total Non-active Active Total

27 1100 total voxels Interpretation 100 voxels have signal (null is false) 80% power -> 80 voxels detected 1000 voxels have no signal (null) 5% type I error -> 50 false positives focus is on controlling this number Declared active Fail to Declare active Total Non-active Active Total

28 Implication of type I error If you run enough tests, you ll find something that is significant This doesn t mean it is truly significant If you run 20 tests with a 5% threshold on type I errors, you expect to have at least 1 significant test This would be a false positive

29 Hypothesis Testing in fmri Mass Univariate Modeling Fit a separate model for each voxel Look at images of statistics Apply Threshold

30 Assessing Statistic Images What threshold will show us signal? High Threshold Med. Threshold Low Threshold t > 5.5 t > 3.5 t > 0.5 Good Specificity Poor Power (risk of false negatives) Poor Specificity (risk of false positives) Good Power

31 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

32 Levels of inference Voxel level Cluster level Peak level Set level

33 Voxel-level Inference Retain voxels above a-level threshold u a Gives best spatial specificity The null hyp. at a single voxel can be rejected Statistic values space

34 Voxel-level Inference Retain voxels above a-level threshold u a Gives best spatial specificity The null hyp. at a single voxel can be rejected u a space

35 Voxel-level Inference Retain voxels above a-level threshold u a Gives best spatial specificity The null hyp. at a single voxel can be rejected u a space Significant Voxels No significant Voxels

36 Cluster-level Inference Two step-process Define clusters by arbitrary threshold u clus u clus space

37 Cluster-level Inference Two step-process Define clusters by arbitrary threshold u clus Retain clusters larger than a-level threshold k a u clus space Cluster not significant k a k a Cluster significant

38 Cluster-level Inference Typically better sensitivity Worse spatial specificity The null hyp. of entire cluster is rejected Only means that one or more of voxels in cluster active u clus space Cluster not significant k a k a Cluster significant

39 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u clus space

40 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 2 Z 3 Z 4 Z5 u clus Z 1 space

41 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u peak Z 2 Z 3 Z 4 Z5 u clus Z 1 space

42 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u peak Z 2 Z 3 Z 4 Z5 u clus Z 1 space

43 Set level inference Is there any activation anywhere in the brain? Omnibus hypothesis test of all voxels, simultaneously If significant, we only know there s activation somewhere in the brain

44 Levels of inference Voxel level Cluster level Peak level Set level

45 Questions for you Why do some approaches require 2 thresholds? What thresholding strategy do people typically use?

46 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

47 What error rate should we control? Per comparison error rate? Family wise error rate? False discovery rate?

48 Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed active FWER Family wise error rate Controls the probability of any false positives Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding

49 Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed active FWER Family wise error rate Controls the probability of any false positives Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding

50 Different types of error rates FDR False discovery rate Of the voxels you deemed significant, what percentage were null

51 FWER FWER vs FDR P(# true null declared active > 1) FDR E (# of true null declared active / # voxels declared active) Declared active Fail to Declare active Total Non-active Active Total

52 FWER FWER vs FDR P(# true null declared active > 1) FDR E (# of true null declared active / # voxels declared active) Declared active Fail to Declare active Total Non-active Active Total

53 False Discovery Rate Illustration: Noise Signal Signal+Noise

54 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

55 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

56 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

57 Considerations with multiple comparisons What statistic you re working with Voxel wise? Cluster wise? What error rate you re controlling Per comparison error rate Family wise error rate False discovery rate

58 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

59 FWER FWER P(# true null declared active > 1) FDR E (# of true null declared active / # voxels declared active) Declared active Fail to Declare active Total Non-active Active Total

60 FWER Correction - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

61 FWER Correction - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

62 FWER Correction - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

63 FWER Correction - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

64 FWER Correction - Bonferroni Can be too conservative Bonferroni assumes all tests are independent fmri data tend to be spatially correlated # of independent tests < # voxels

65 Smooth data How will the Bonferroni correction work with smoothed data? Will false positive rate increase or decrease?

66 FWER Random Field theory Parametric approach to controlling false positives Parametric = there s an equation that will spit out the p-value Beyond the scope of this course Tends to be as conservative as Bonferroni

67 FWER Random Field theory Parametric approach to controlling false positives Parametric = there s an equation that will spit out the p-value Beyond the scope of this course Tends to be as conservative as Bonferroni

68 FWER Random Field theory Parametric approach to controlling false positives Parametric = there s an equation that will spit out the p-value Voxelwise version tends to be as conservative as Bonferroni

69 Another way to control all tests Statistic Magnitude Test Number

70 Another way to control all tests Statistic Magnitude Test Number

71 Another way to control all tests Statistic Magnitude Test Number

72 Another way to control all tests Statistic Magnitude Test Number

73 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

74 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

75 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

76 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

77 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

78 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a

79 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a

80 FWER with max statistic FWER & distribution of maximum FWER = P(FWE) = P(One or more voxels ³ u H o ) = P(Max voxel ³ u H o ) 100(1-a)%ile of max dist n controls FWER FWER = P(Max voxel ³ u a H o ) a u a a

81 FWER MTP Solutions: Random Field Theory Euler Characteristic c u Topological Measure No holes Never more than 1 blob #blobs - #holes At high thresholds, just counts blobs Random Field FWER = P(Max voxel ³ u H o ) = P(One or more blobs H o )» P(c u ³ 1 H o )» E(c u H o ) Threshold Suprathreshold Sets

82 Distribution details Math is hairy! Nichols and Hayasaka 2003 Cao and Worsley 2001 What you need to know Depends on smoothness of your image Must quantify smoothness and it is important to report when using RFT

83 General idea E(c u )» Mathy stuff *Volume/Smoothness We know what the volume is What is smoothness?

84 Smoothness How smooth are the data? Measured by FWHM=[FWHM x, FWHM y, FWHM z ] Starting with white noise smooth with a gaussian How large does the variance of that gaussian need to be such that the smoothness matches your data?

85 RESEL RESolution Element RESEL=FWHM x x FWHM y x FWHM z RESEL count If your voxels were the size of a RESEL, how many are required to fill your volume? 10 voxels, 2.5 voxel FWHM smoothness Þ 4 RESELS

86 voxels FWHM= 2.5 voxels RESEL count=4

87 Note about RESEL count Not the number of independent tests Not the magic bullet for a better Bonferroni Re-expression of volume in terms of smoothness We need it, since it is necessary to calculate our p-values

88 Revisit distribution E(c u )» Mathy stuff *Volume/Smoothness Smoothness is defined in RESELs E(c u ) is our p-value How does a p-value change as volume increases? How does a p-value change as smoothness increases?

89 RFT adapts For larger volumes it is more strict Multiple comparison problem is worse For smoother data it is less strict Multiple comparison problem is less severe

90 Shortcomings of RFT Requires estimating a lot of parameters Random field must be sufficiently smooth If you don t spatially smooth the data enough, RFT doesn t work well I ll cover the Eklund paper later on today!

91 Bonferroni and RFT u RF = 9.87 u Bonf = sig. vox. t 11 Statistic, RF & Bonf. Threshold

92 RFT Voxelwise RFT is rarely used in practice Too conservative Cluster wise RFT is very common We ll learn about cluster stats with permutation testing

93 FYI If you re using RFT, you probably shouldn t lower the cluster forming threshold Assumptions could break down If you really want to lower it, switch to nonparameteric approaches SnPM Randomise

94 Questions for you Why do we use the max statistic for multiple comparison correction? Was this a voxelwise or clusterwise approach?

95 Parametric vs Nonparametric Parametric Assume distribution shape Typically 1 or more parameters must be estimated Nonparametric No assumption on distribution shape Use data to construct distribution Related to bootstrap and jackknife, BUT not the same!!!

96 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

97 Permutation test Generally can be used when the true distribution shape is unknown Data don t follow a normal distribution Generally doesn t control for multiple comparisons Using in conjunction with the max statistic tackles 2 problems Not knowing the structure of the distribution Control FWER

98 Permutation test Generally can be used when the true distribution shape is unknown Data don t follow a normal distribution Generally doesn t control for multiple comparisons Using in conjunction with the max statistic tackles 2 problems Not knowing the structure of the distribution Control FWER

99 Permutation test Generally can be used when the true distribution shape is unknown Data don t follow a normal distribution Generally doesn t control for multiple comparisons Using in conjunction with the max statistic tackles 2 problems Not knowing the structure of the distribution Control FWER

100 Permutation test Without using max statistic So we understand how it generally works With max statistic So we understand how to control FWER

101 Permutation test Parametric methods Assume distribution of statistic under null hypothesis Nonparametric methods Use data to find distribution of statistic under null hypothesis Any statistic! 5% Parametric Null Distribution 5% Nonparametric Null Distribution

102 Permutation Test Toy Example Data from voxel in visual stim. experiment A: Active, flashing checkerboard B: Baseline, fixation 6 blocks, ABABAB Just consider block averages... A B A B A B Null hypothesis H o No experimental effect, A & B labels arbitrary Statistic Mean difference

103 Permutation Test Toy Example Under H o Consider all equivalent relabelings AAABBB ABABAB BAAABB BABBAA AABABB ABABBA BAABAB BBAAAB AABBAB ABBAAB BAABBA BBAABA AABBBA ABBABA BABAAB BBABAA ABAABB ABBBAA BABABA BBBAAA

104 Permutation Test Toy Example Under H o Consider all equivalent relabelings Compute all possible statistic values AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

105 Permutation Test Toy Example Under H o Consider all equivalent relabelings Compute all possible statistic values Find 95%ile of permutation distribution AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

106 Permutation Test Toy Example Under H o Consider all equivalent relabelings Compute all possible statistic values Find 95%ile of permutation distribution AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

107 Permutation Test Toy Example Under H o Consider all equivalent relabelings Compute all possible statistic values Find 95%ile of permutation distribution

108 Small Sample Sizes Permutation test doesn t work well with small sample sizes Possible p-values for previous example: 0.05, 0.1, 0.15, 0.2, etc Tends to be conservative for small sample sizes

109 Permutation Test & Exchangeability Exchangeability is fundamental Def: Distribution of the data unperturbed by permutation Under H 0, exchangeability justifies permuting data Allows us to build permutation distribution

110 What is exchanged? Under the null, what can be swapped? Null data (slope = 0) x y

111 Null data (slope = 0) x y What is exchanged? Under the null, what can be swapped?

112 What is exchanged? Under the null, what can be swapped? Null data (slope = 0) x y

113 Original data y 1 y 2 y 3 y 4 y 5 y 6 y 7 y 8 x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8

114 Permuted data y 8 y 3 y 2 y 7 y 6 y 1 y 4 y 5 x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8

115 When exchangeability doesn t hold Correlated data Temporal correlation Unpermuted Permuted

116 When exchangeability doesn t hold Correlated data Family data Unpermuted Permuted

117 When exchangeability doesn t hold Influential outliers with a continuous covariate Original Data (null) Y Y X Permutation Permuted X

118 When exchangeability does hold Independent subjects Homoscedasticity Heteroscedasticity is fine if you re running a 1- sample t-test (I m sure there are exceptions)

119 Permutation Test & Exchangeability Subjects are exchangeable Under Ho, each subject s A/B labels can be flipped fmri scans are not exchangeable under Ho If no signal, can we permute over time? No, permuting disrupts order, temporal autocorrelation

120 Permutation Test & Exchangeability Two sample t test Compare subjects in group 1 to subjects in group 2 Randomly assign group labels in permutations One sample t test Randomly flip sign of values for some subjects Correlation Randomly reorder subjects in dependent variable

121 Questions for you Why are small sample sizes problematic for permutation testing?

122 Controlling FWER: Permutation Test Parametric methods Assume distribution of max statistic under null hypothesis Nonparametric methods Use data to find distribution of max statistic under null hypothesis Again, any max statistic! 5% Parametric Null Max Distribution 5% Nonparametric Null Max Distribution

123 Permutation Test Other Statistics Collect max distribution To find threshold that controls FWER Consider smoothed variance t statistic To regularize low-df variance estimate

124 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

125 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

126 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

127 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

128 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

129 Max statistic for imaging data 1. Compute your statistics map for original data 2. Shuffle labels and compute statistics map 3. Save the largest statistic over the whole brain 4. Repeat steps 2-3 many times ( ,000) 5. Use distribution of stats over permutations to compute threshold 6. Apply threshold to map from step 1

130 x y x y x y3 Original Data Null Distribution Maximum T Frequency

131 Null Distribution Maximum T Frequency x y1 T= x y2 T= x y3 T= 1.55 Permutation 1

132 Null Distribution Maximum T Frequency x y1 T= x y2 T= x y3 T=1.45 Permutation 2

133 Null Distribution Maximum T Frequency x y1 T= x y2 T= x y3 T= 0.36 Permutation 100

134 Null Distribution Maximum T Frequency x y1 T= x y2 T= x y3 T= 2.25 Permutation 5000 p=0.39 p=0.21 p=0.003

135 Permutation Test Smoothed Variance t Collect max distribution To find threshold that controls FWER Consider smoothed variance t statistic mean difference variance t-statistic

136 Permutation Test Smoothed Variance t Collect max distribution To find threshold that controls FWER Consider smoothed variance t statistic mean difference smoothed variance Smoothed Variance t-statistic

137 Permutation Test Example fmri Study of Working Memory Active subjects, block design Marshuetzet al (2000) Item Recognition Active: View five letters, 2s pause, view probe letter, respond Baseline: View XXXXX, 2s pause, view Y or N, respond Second Level RFX Difference image, A-B constructed for each subject One sample t test UBKDA... Baseline XXXXX... D N yes... no

138 Permutation Test Example Permute! 2 12 = 4,096 ways to flip 12 A/B labels For each, note maximum of t image. Permutation Distribution Maximum t Maximum Intensity Projection Thresholded t

139 u Perm = sig. vox. t 11 Statistic, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Statistic, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Statistic, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

140 u Perm = sig. vox. t 11 Statistic, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Statistic, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Statistic, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

141 u Perm = sig. vox. t 11 Statistic, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Statistic, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Statistic, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

142 u Perm = sig. vox. t 11 Statistic, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Statistic, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Statistic, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

143 Permutation test cluster statistic Two step-process Define clusters by arbitrary threshold u clus u clus space

144 Permutation test cluster statistic Two step-process Define clusters by arbitrary threshold u clus Retain clusters larger than a-level threshold k a u clus space Cluster not significant k a k a Cluster significant

145 Permutation test cluster statistics Cluster size Simply count how many voxels are in the cluster Cluster mass Sum up the statistics in the cluster

146 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

147 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

148 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

149 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

150 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

151 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

152 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold

153 Permutation test cluster statistics 1. Find clusters with original data 2. Permute labels 3. Compute statistics 4. Apply cluster-forming threshold 5. Compute cluster statistics 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many times ( ,000) 8. Use distribution from step 7 to find cluster (size or mass) threshold & apply to step 1

154 Questions for you Why don t permutation tests, alone, fix multiple comparisons? What did we need to use to address multiple comparisons? How are the voxelwise and clusterwise permutation tests set up?

155 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

156 FWER FWER vs FDR P(# true null declared active > 1) FDR E (# of true null declared active / # voxels declared active) Declared active Declared inactive Total Non-active Active Total

157 Controlling FDR Tends to be less conservative than controlling FWER What rate is appropriate? Imagers use 5%...out of habit FDR people I ve met outside of imaging often use higher values Decide before you threshold your data Don t choose what makes your data look good

158 Benjamini & Hochberg Procedure Select desired limit a on FDR Order p-values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v a 1 Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 0 i/v 1

159 Benjamini & Hochberg Procedure Select desired limit a on FDR Order p-values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v a 1 Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 0 i/v i/v a 1

160 FDR Example FWER Perm. Thresh. = voxels FDR Threshold = ,073 voxels

161 Where we re going Review of hypothesis testing introduce multiple testing problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family-wise error control approaches (parametric/nonparametric) FDR Relating all of this to SPM output

162 Guess what? Now you have the knowledge needed to understand a huge/daunting table SPM spits out! Let s do it

163 SPM output

164 Which level of inference is missing? SPM output

165 what exciting conclusion can we make? SPM output

166 Recall: FWE correction shown earlier was super conservative compared to FDR. Why does this look different? SPM output

167 SPM output What do you think K E is? What statistic does the p-value correspond to?

168 The uncorrected stat doesn t take the search volume into account SPM output

169 See the note at the bottom? SPM output

170 Do any clusters have more than one peak? SPM output

171 SPM output Last, but not least, you ll use this in lab. This is used to threshold clusters so you can look at only the significant ones

172 SPM output Compare this threshold to the FWE p-values for cluster stats

173 Questions? That s it!