Controlling for mul-ple comparisons in imaging analysis. Wednesday, Lecture 2 Jeane:e Mumford University of Wisconsin - Madison

Transcription

1 Controlling for mul-ple comparisons in imaging analysis Wednesday, Lecture 2 Jeane:e Mumford University of Wisconsin - Madison

2 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

3 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

4 Review of hypothesis tes-ng What is H0? What is HA? What are the steps of carrying out a hypothesis test?

5 Review of hypothesis tes-ng What is H0? What is HA? What are the steps of carrying out a hypothesis test?

6 Steps of hypothesis tes-ng

7 Steps of hypothesis tes-ng

8 Steps of hypothesis tes-ng

9 Steps of hypothesis tes-ng What do we compare this area to (p- value)?

10 What does the p- value mean? p = 0.01

11 What does the p- value mean? p = 0.01 If the null distribu-on is true

12 What does the p- value mean? p = 0.01 If the null distribu-on is true The probability of observing my sta-s-c (or something more extreme than it) is 0.01

13 What does the p- value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

14 What does the p- value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

15 What does the p- value threshold We choose 0.05 imply? Less than 0.05 and we reject the null hypothesis Greater than 0.05 and we fail to reject the null hypothesis

16 Type I error Assuming the null is true, the probability that we reject the null

17 Type I error Assuming the null is true, the probability that we reject the null 5% of the -me, we ll have a false posi-ve

18 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 Interpreta-on 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

19 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 Interpreta-on 5% type I error - > 50 false posi-ves What we know (test results) Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

20 1100 total voxels 100 voxels have β=δ 80% power - > 80 voxels detected 1000 voxels have β=0 Interpreta-on 5% type I error - > 50 false posi-ves What we don t know (truth) Non- ac-ve Ac-ve Total Declared ac-ve Declared inac-ve Total

21 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve 1000 Ac-ve 100 Total 1100

22 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve 1000 Ac-ve Total 1100

23 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve 1000 Ac-ve 80 (Power) 20 (Type II err.) 100 Total 1100

24 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve 80 (Power) 20 (Type II err.) 100 Total 1100

25 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve 50 (Type I err.) 950 (Correct) 1000 Ac-ve 80 (Power) 20 (Type II err.) 100 Total 1100

26 Interpreta-on 1100 total voxels 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

27 1100 total voxels Interpreta-on 100 voxels have signal (null is false) 80% power - > 80 voxels detected 1000 voxels have no signal (null) 5% type I error - > 50 false posi-ves focus is on controlling this number Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

28 Implica-on of type I error If you run enough tests, you ll find something that is significant This doesn t mean it is truly significant If you run 20 tests with a 5% threshold on type I errors, you expect to have at least 1 significant test This would be a false posi-ve

29 Hypothesis Tes-ng in fmri Mass Univariate Modeling Fit a separate model for each voxel Look at images of sta-s-cs Apply Threshold

30 Assessing Sta-s-c Images What threshold will show us signal? High Threshold Med. Threshold Low Threshold t > 5.5 t > 3.5 t > 0.5 Good Specificity Poor Power (risk of false negatives) Poor Specificity (risk of false positives) Good Power

31 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

32 Levels of inference Voxel level Cluster level Peak level Set level

33 Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa-al specificity The null hyp. at a single voxel can be rejected Statistic values space

34 Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa-al specificity The null hyp. at a single voxel can be rejected u α space

35 Voxel- level Inference Retain voxels above α- level threshold u α Gives best spa-al specificity The null hyp. at a single voxel can be rejected u α space Significant Voxels No significant Voxels

36 Cluster- level Inference Two step- process Define clusters by arbitrary threshold u clus u clus space

37 Cluster- level Inference Two step- process Define clusters by arbitrary threshold u clus Retain clusters larger than α- level threshold k α u clus space Cluster not significant k α k α Cluster significant

38 Cluster- level Inference Typically be:er sensi-vity Worse spa-al specificity The null hyp. of en-re cluster is rejected Only means that one or more of voxels in cluster ac-ve u clus space Cluster not significant k α k α Cluster significant

39 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u clus space

40 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 Z 2 Z 3 u clus Z 1 Z 5 space

41 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space

42 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space

43 Set level inference Is there any ac-va-on anywhere in the brain? Omnibus hypothesis test of all voxels, simultaneously If significant, we only know there s ac-va-on somewhere in the brain

44 Levels of inference Voxel level Cluster level Peak level Set level

45 Ques-ons for you Why do some approaches require 2 thresholds? What thresholding strategy do people typically use?

46 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

47 What error rate should we control? Per comparison error rate? Family wise error rate? False discovery rate?

48 Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed ac-ve FWER Family wise error rate Controls the probability of any false posi-ves Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding

49 Different types of error rates PCER Per comparison error rate Controlling each voxel at 5% Expect 5% of null voxels will be (mistakenly) deemed ac-ve FWER Family wise error rate Controls the probability of any false posi-ves Run 20 NULL group analyses (on 20 data sets) and only 1 analysis will have a significant finding

50 Different types of error rates FDR False discovery rate Of the voxels you deemed significant, what percentage were null

51 FWER FWER vs FDR P(# true null declared ac-ve > 1) FDR E (# of true null declared ac-ve / # voxels declared ac-ve) Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

52 FWER FWER vs FDR P(# true null declared ac-ve > 1) FDR E (# of true null declared ac-ve / # voxels declared ac-ve) Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

53 False Discovery Rate Illustra-on: Noise Signal Signal+Noise

54 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

55 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

56 Control of Per Comparison Rate at 10% 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Control of Familywise Error Rate at 10% Occurrence of Familywise Error FWE Control of False Discovery Rate at 10% 6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7% Percentage of Activated Pixels that are False Positives

57 Considera-ons with mul-ple comparisons What sta-s-c you re working with Voxel wise? Cluster wise? What error rate you re controlling Per comparison error rate Family wise error rate False discovery rate

58 Correlated data Images typically have correlated voxels # of false posi-ves = 0.05 x (# of independent tests) Extreme example Data are smoothed so much all voxels are iden-cal Only 1 out of 20 data sets would have a false posi-ve

59 Correlated data Images typically have correlated voxels # of false posi-ves = 0.05 x (# of independent tests) Extreme example Data are smoothed so much all voxels are iden-cal Only 1 out of 20 data sets would have a false posi-ve

60 Correlated data Coun-ng false posi-ves becomes tricky, since you don t know the number of independent things

61 When data are not correlated P- values computed from simulated null data

62 When data are not correlated Thresholded p< > 4.7% are false posi-ves

63 Correlated data Coun-ng false posi-ves becomes tricky, since you don t know the number of independent things

64 Same demo, with smoothed data Thresholded p- value map - > 4.2% are FP

65 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

66 FWER FWER P(# true null declared ac-ve > 1) FDR E (# of true null declared ac-ve / # voxels declared ac-ve) Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

67 FWER Correc-on - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

68 FWER Correc-on - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

69 FWER Correc-on - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

70 FWER Correc-on - Bonferroni Based on the Bonferroni inequality P (E 1 or E 2 or...e n ) apple nx i=1 P (E i ) If P (Y i passes H 0 ) apple /n then nx P (some Y i passes H 0 ) apple P (Y i passes H 0 ) apple i=1 For 100,000 voxels = 0.05/100, 000 =

71 FWER Correc-on - Bonferroni Can be too conserva-ve Bonferroni assumes all tests are independent fmri data tend to be spa-ally correlated # of independent tests < # voxels

72 Smooth data How will the Bonferroni correc-on work with smoothed data? Will false posi-ve rate increase or decrease?

73 Ques-ons Why doesn t Bonferroni work well with our imaging data? Why does smoothness make mul-ple comparison correc-on more tricky?

74 FWER Random Field theory Parametric approach to controlling false posi-ves Parametric = there s an equa-on that will spit out the p- value Beyond the scope of this course Tends to be as conserva-ve as Bonferroni

75 FWER Random Field theory Parametric approach to controlling false posi-ves Parametric = there s an equa-on that will spit out the p- value Beyond the scope of this course Tends to be as conserva-ve as Bonferroni

76 FWER Random Field theory Parametric approach to controlling false posi-ves Parametric = there s an equa-on that will spit out the p- value Voxelwise version tends to be as conserva-ve as Bonferroni

77 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

78 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

79 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

80 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

81 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

82 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α

83 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α

84 FWER with max sta-s-c FWER & distribu-on of maximum FWER = P(FWE) = P(One or more voxels u H o ) = P(Max voxel u H o ) 100(1- α)%ile of max dist n controls FWER FWER = P(Max voxel u α H o ) α u α α

85 FWER MTP Solu-ons: Random Field Theory Euler Characteris-c χ u Topological Measure No holes Never more than 1 blob #blobs - #holes At high thresholds, just counts blobs Random Field FWER = P(Max voxel u H o ) = P(One or more blobs H o ) P(χ u 1 H o ) E(χ u H o ) Threshold Suprathreshold Sets

86 Distribution details Math is hairy! Nichols and Hayasaka 2003 Cao and Worsley 2001 What you need to know Depends on smoothness of your image Must quantify smoothness and it is important to report when using RFT

87 General idea E(χ u ) Mathy stuff *Volume/Smoothness We know what the volume is What is smoothness?

88 Smoothness How smooth are the data? Measured by FWHM=[FWHM x, FWHM y, FWHM z ] Starting with white noise smooth with a gaussian How large does the variance of that gaussian need to be such that the smoothness matches your data?

89 RESEL RESolution Element RESEL=FWHM x x FWHM y x FWHM z RESEL count If your voxels were the size of a RESEL, how many are required to fill your volume? 10 voxels, 2.5 voxel FWHM smoothness 4 RESELS

90 voxels FWHM= 2.5 voxels RESEL count=4

91 Note about RESEL count Not the number of independent tests Not the magic bullet for a better Bonferroni Re-expression of volume in terms of smoothness We need it, since it is necessary to calculate our p-values

92 Revisit distribution E(χ u ) Mathy stuff *Volume/Smoothness Smoothness is defined in RESELs E(χ u ) is our p-value How does a p-value change as volume increases? How does a p-value change as smoothness increases?

93 RFT adapts For larger volumes it is more strict Multiple comparison problem is worse For smoother data it is less strict Multiple comparison problem is less severe

94 Shortcomings of RFT Requires estimating a lot of parameters Random field must be sufficiently smooth If you don t spatially smooth the data enough, RFT doesn t work well I ll cover the Eklund paper later on today!

95 Bonferroni and RFT u RF = 9.87 u Bonf = sig. vox. t 11 Sta-s-c, RF & Bonf. Threshold

96 RFT Voxelwise RFT is rarely used in prac-ce Too conserva-ve Cluster wise RFT is very common We ll learn about cluster stats with permuta-on tes-ng

97 FYI If you re using RFT, you probably shouldn t lower the cluster forming threshold Assump-ons could break down If you really want to lower it, switch to nonparameteric approaches SnPM Randomise

98 Ques-ons for you Why do we use the max sta-s-c for mul-ple comparison correc-on? Was this a voxelwise or clusterwise approach?

99 Parametric vs Nonparametric Parametric Assume distribu-on shape Typically 1 or more parameters must be es-mated Nonparametric No assump-on on distribu-on shape Use data to construct distribu-on Related to bootstrap and jackknife, BUT not the same!!!

100 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

101 Permuta-on test Generally can be used when the true distribu-on shape is unknown Data don t follow a normal distribu-on Generally doesn t control for mul-ple comparisons Using in conjunc-on with the max sta-s-c tackles 2 problems Not knowing the structure of the distribu-on Control FWER

102 Permuta-on test Generally can be used when the true distribu-on shape is unknown Data don t follow a normal distribu-on Generally doesn t control for mul-ple comparisons Using in conjunc-on with the max sta-s-c tackles 2 problems Not knowing the structure of the distribu-on Control FWER

103 Permuta-on test Generally can be used when the true distribu-on shape is unknown Data don t follow a normal distribu-on Generally doesn t control for mul-ple comparisons Using in conjunc-on with the max sta-s-c tackles 2 problems Not knowing the structure of the distribu-on Control FWER

104 Permuta-on test Without using max sta-s-c So we understand how it generally works With max sta-s-c So we understand how to control FWER

105 Permuta-on test Parametric methods Assume distribu-on of sta-s-c under null hypothesis Nonparametric methods Use data to find distribu-on of sta-s-c under null hypothesis Any sta-s-c! 5% Parametric Null Distribu-on 5% Nonparametric Null Distribu-on

106 Permuta-on Test Toy Example Data from voxel in visual s-m. experiment A: Ac-ve, flashing checkerboard B: Baseline, fixa-on 6 blocks, ABABAB Just consider block averages... A B A B A B Null hypothesis H o No experimental effect, A & B labels arbitrary Sta-s-c Mean difference

107 Permuta-on Test Toy Example Under H o Consider all equivalent relabelings AAABBB ABABAB BAAABB BABBAA AABABB ABABBA BAABAB BBAAAB AABBAB ABBAAB BAABBA BBAABA AABBBA ABBABA BABAAB BBABAA ABAABB ABBBAA BABABA BBBAAA

108 Permuta-on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta-s-c values AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

109 Permuta-on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta-s-c values Find 95%ile of permuta-on distribu-on AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

110 Permuta-on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta-s-c values Find 95%ile of permuta-on distribu-on AAABBB 4.82 ABABAB 9.45 BAAABB BABBAA AABABB ABABBA 6.97 BAABAB 1.10 BBAAAB 3.15 AABBAB ABBAAB 1.38 BAABBA BBAABA 0.67 AABBBA ABBABA BABAAB BBABAA 3.25 ABAABB 6.86 ABBBAA 1.48 BABABA BBBAAA -4.82

111 Permuta-on Test Toy Example Under H o Consider all equivalent relabelings Compute all possible sta-s-c values Find 95%ile of permuta-on distribu-on

112 Small Sample Sizes Permutation test doesn t work well with small sample sizes Possible p-values for previous example: 0.05, 0.1, 0.15, 0.2, etc Tends to be conservative for small sample sizes

113 Permuta-on Test & Exchangeability Exchangeability is fundamental Def: Distribu-on of the data unperturbed by permuta-on Under H 0, exchangeability jus-fies permu-ng data Allows us to build permuta-on distribu-on

114 Permuta-on Test & Exchangeability Subjects are exchangeable Under Ho, each subject s A/B labels can be flipped fmri scans are not exchangeable under Ho If no signal, can we permute over -me? No, permu-ng disrupts order, temporal autocorrela-on

115 Permuta-on Test & Exchangeability Two sample t test Compare subjects in group 1 to subjects in group 2 Randomly assign group labels in permuta-ons One sample t test Randomly flip sign of values for some subjects

116 Ques-ons for you What is permuted for a 1- sample t- test? What is permuted for a 2- sample t- test? What is permuted for a correla-on? Why are small sample sizes problema-c for permuta-on tes-ng?

117 Controlling FWER: Permuta-on Test Parametric methods Assume distribu-on of max sta-s-c under null hypothesis Nonparametric methods Use data to find distribu-on of max sta-s-c under null hypothesis Again, any max sta-s-c! 5% Parametric Null Max Distribu-on 5% Nonparametric Null Max Distribu-on

118 Permuta-on Test Other Sta-s-cs Collect max distribu-on To find threshold that controls FWER Consider smoothed variance t sta-s-c To regularize low- df variance es-mate

119 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

120 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

121 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

122 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

123 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

124 Max sta-s-c for imaging data 1. Compute your sta-s-cs map for original data 2. Shuffle labels and compute sta-s-cs map 3. Save the largest sta-s-c over the whole brain 4. Repeat steps 2-3 many -mes ( ,000) 5. Use distribu-on of stats over permuta-ons to compute threshold 6. Apply threshold to map from step 1

125 Permuta-on Test Smoothed Variance t Collect max distribu-on To find threshold that controls FWER Consider smoothed variance t sta-s-c mean difference variance t-statistic

126 Permuta-on Test Smoothed Variance t Collect max distribu-on To find threshold that controls FWER Consider smoothed variance t sta-s-c mean difference smoothed variance Smoothed Variance t-statistic

127 Permuta-on Test Example fmri Study of Working Memory 12 subjects, block design Marshuetz et al (2000) Item Recogni-on Ac-ve: View five le:ers, 2s pause, view probe le:er, respond Baseline: View XXXXX, 2s pause, view Y or N, respond Second Level RFX Difference image, A- B constructed for each subject One sample t test Active D UBKDA yes Baseline N XXXXX no

128 Permuta-on Test Example Permute! 2 12 = 4,096 ways to flip 12 A/B labels For each, note maximum of t image. Permuta-on Distribu-on Maximum t Maximum Intensity Projec-on Thresholded t

129 u Perm = sig. vox. t 11 Sta-s-c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta-s-c, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Sta-s-c, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

130 u Perm = sig. vox. t 11 Sta-s-c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta-s-c, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Sta-s-c, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

131 u Perm = sig. vox. t 11 Sta-s-c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta-s-c, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Sta-s-c, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

132 u Perm = sig. vox. t 11 Sta-s-c, Nonparametric Threshold u RF = 9.87 u Bonf = sig. vox. t 11 Sta-s-c, RF & Bonf. Threshold RFT threshold is conservative (not smooth enough, d.f. too small) 378 sig. vox. Smoothed Variance t Sta-s-c, Nonparametric Threshold Permutation test is more efficient than Bonferroni since it accounts for smoothness Smooth variance is more efficient for small d.f.

133 Permuta-on test cluster sta-s-c Two step- process Define clusters by arbitrary threshold u clus u clus space

134 Permuta-on test cluster sta-s-c Two step- process Define clusters by arbitrary threshold u clus Retain clusters larger than α- level threshold k α u clus space Cluster not significant k α k α Cluster significant

135 Permuta-on test cluster sta-s-cs Cluster size Simply count how many voxels are in the sta-s-c Cluster mass Sum up the sta-s-cs in the cluster

136 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

137 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

138 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

139 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

140 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

141 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

142 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold

143 Permuta-on test cluster sta-s-cs 1. Find clusters with original data 2. Permute labels 3. Compute sta-s-cs 4. Apply cluster- forming threshold 5. Compute cluster sta-s-cs 6. Save largest (cluster size or mass) 7. Repeat steps 2-3 many -mes ( ,000) 8. Use distribu-on from step 7 to find cluster (size or mass) threshold & apply to step 1

144 Ques-ons for you Why don t permuta-on tests, alone, fix mul-ple comparisons? What did we need to use to address mul-ple comparisons? How are the voxelwise and clusterwise permuta-on tests set up?

145 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

146 FWER FWER vs FDR P(# true null declared ac-ve > 1) FDR E (# of true null declared ac-ve / # voxels declared ac-ve) Declared ac-ve Declared inac-ve Total Non- ac-ve Ac-ve Total

147 Controlling FDR Tends to be less conserva-ve than controlling FWER What rate is appropriate? Imagers use 5%...out of habit FDR people I ve met outside of imaging ozen use higher values Decide before you threshold your data Don t choose what makes your data look good

148 Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α 1 Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 0 i/v 1

149 Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α 1 Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 0 i/v i/v α 1

150 Benjamini & Hochberg Procedure Select desired limit α on FDR Order p- values, p (1) p (2)... p (v) Let r be largest i such that p (i) i/v α 1 Reject all hypotheses corresponding to p (1),..., p (r). p-value p (i) 0 0 i/v α i/v 1

151 FDR Example FWER Perm. Thresh. = voxels FDR Threshold = ,073 voxels

152 Where we re going Review of hypothesis tes-ng introduce mul-ple tes-ng problem Levels of inference (voxel/cluster/peak/set) Types of error rate control (none/fwer/fdr) Family- wise error control approaches (parametric/ nonparametric) FDR Rela-ng all of this to SPM output

153 Guess what? Now you have the knowledge needed to understand a huge/daun-ng table SPM spits out! Let s do it

154 SPM output

155 Which level of inference is missing? SPM output

156 what exci-ng conclusion can we make? SPM output

157 Recall: FWE correc-on shown earlier was super conserva-ve compared to FDR. Why does this look different? SPM output

158 SPM output What do you think K E is? What sta-s-c does the p- value correspond to?

159 The uncorrected stat doesn t take the search volume into account SPM output

160 See the note at the bo:om? SPM output

161 Do any clusters have more than one peak? SPM output

162 SPM output Last, but not least, you ll use this in lab. This is used to threshold clusters so you can look at only the significant ones

163 SPM output Compare this threshold to the FWE p- values for cluster stats

164 Ques-ons? That s it!