Power analysis. Wednesday, Lecture 3 Jeanette Mumford University of Wisconsin - Madison

Transcription

1 Power analysis Wednesday, Lecture 3 Jeanette Mumford University of Wisconsin - Madison

2 Power Analysis-Why? To answer the question. How many subjects do I need for my study? How many runs per subject should I collect? To create thorough grant applications that will make reviewers happy! Don t waste money on underpowered studies OR collecting data on more subjects than you need

3 Wait, what is efficiency? Remember efficiency? What specifically did we look at? What would happen with efficiency if we simply kept adding subjects?

4 Power More meaningful than efficiency We know that a design with over 80% power is a pretty good one Important point With efficiency we were optimizing the time series model With power we are focusing on the group analysis

5 What is Power? Null Distribu3on Alterna3ve Distribu3on Power: The probability 0.4 of rejecting H 0 when 0.35 H A is true 0.3 Specify your null 0.25 distribution Mean=0, variance=σ Specify the effect size 0.15 (Δ), which leads to 0.1 alternative distribution 0.05 Specify the false α Power positive rate, α Δ/σ

6 Interpreta3on 1100 total voxels 100 voxels have β=δ A test with 50% power on average will detect 50 of these voxels with true ac3va3on 1000 voxels have β=0 α=5% implies on average 50 null voxels will have false posi3ves fmri Power Test Result (observed) Reject H 0 Accept H 0 Truth (unobserved) H 0 True Type I Error α 50 Correct 950 H 0 False Power 50 Type II Error

7 Two approaches for trea3ng the data Region of interest Focus on a set of voxels Whole brain Focus on all voxels, simultaneously

8 Region of interest analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N...

9 Region of interest analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N... Subject-specific ac3va3on maps (3D)

10 Region of interest analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N... Average data over region of interest

11 Region of interest analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N... Average data over region of interest (1D) Single region of interest analysis Brain ac3va3on Depression

12 Whole brain analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N...

13 Whole brain analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N... Subject-specific ac3va3on maps (3D)

14 Whole brain analysis Subject-specific 3me series data (4D) subject 1 subject 2 subject N... Subject-specific ac3va3on maps (3D) Group sta3s3c map (3D)

15 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 at each voxel Family wise error Probability of any false posi3ves False discovery rate Propor3on of voxels found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

16 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 for each sta3s3c Family wise error Probability of any false posi3ves False discovery rate Propor3on of voxels found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

17 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 for each sta3s3c Family wise error Probability of any false posi3ves False discovery rate Propor3on of voxels found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

18 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 for each sta3s3c Family wise error Probability of any false posi3ves False discovery rate Propor3on of sta3s3cs found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

19 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 for each sta3s3c Family wise error Probability of any false posi3ves False discovery rate Propor3on of sta3s3cs found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

20 Voxel-level Inference Retain voxels above α-level threshold u α Gives best spa3al specificity The null hyp. at a single voxel can be rejected Statistic values space

21 Voxel-level Inference Retain voxels above α-level threshold u α Gives best spa3al specificity The null hyp. at a single voxel can be rejected u α space

22 Voxel-level Inference Retain voxels above α-level threshold u α u α space Significant Voxels No significant Voxels

23 Cluster-level Inference Two step-process Define clusters by arbitrary threshold u clus u clus space

24 Cluster-level Inference Two step-process Define clusters by arbitrary threshold u clus Retain clusters larger than α-level threshold k α u clus space Cluster not significant k α k α Cluster significant

25 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors u clus space

26 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 Z 2 Z 3 u clus Z 1 Z 5 space

27 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Similarly to cluster level inference, significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space

28 Peak level inference Again start with a cluster forming threshold Instead of cluster size, focus on peak height Significance applies to a set of voxels The peak and its neighbors Z 4 u peak Z u 1 clus Z 2 Z 3 Z 5 space

29 Whole brain: Thresholding the map What type of error rate does the researcher want to control? Per comparison error Use p < 0.05 for each sta3s3c Family wise error Probability of any false posi3ves False discovery rate Propor3on of sta3s3cs found to be ac3ve that are false ac3va3ons What sta3s3c does the researcher want to use? Voxelwise Clusterwise Peakwise

30 Power analysis Are studies currently well powered? How well do the methods meet the needs of researchers

31 Sample sizes over 3me Poldrack et al. (2016) Scanning the Horizon: Future Challenges of NeuroImaging Research. BioRxiv preprint

32 Sample sizes over 3me Poldrack et al. (2016) Scanning the Horizon: Future Challenges of NeuroImaging Research. BioRxiv preprint

33 2002: Desmond and Glover ROI analysis approach Simplified 3me series model Based on mixed model Abstract says about 12 subjects for 80% power Only the discussion points out different studies could have smaller effect sizes Desmond and Glover (2002) PMID:

34 2002: Desmond and Glover ROI analysis approach Simplified 3me series model Based on mixed model Abstract says about 12 subjects for 80% power Only the discussion points out different studies could have smaller effect sizes Desmond and Glover (2002) PMID:

35 2002: Desmond and Glover ROI analysis approach Simplified 3me series model Based on mixed model Abstract says about 12 subjects for 80% power Only the discussion points out different studies could have smaller effect sizes Desmond and Glover (2002) PMID:

36 2002: Desmond and Glover ROI analysis approach Simplified 3me series model Based on mixed model Abstract says about 12 subjects for 80% power Only the discussion points out different studies could have smaller effect sizes Desmond and Glover (2002) PMID:

37 2006: Mumford and Nichols ROI-based Match the true models used more closely Desmond and Glover were likely overes3ma3ng power

38 2006: Mumford and Nichols ROI-based Match the true models used more closely Desmond and Glover were likely overes3ma3ng power

39 2006: Mumford and Nichols ROI-based Match the true models used more closely Previous approach was an overes3mate Desmond & Glover Correct

40 2006: Mumford and Nichols ROI-based Match the true models used more closely Previous approach was an overes3mate MATLAB toolbox fmripower.org Desmond & Glover Correct

41 fmripower Beta version at fmripower.org ROI based power analysis Can apply to old FSL or SPM anlayses Runs in Matlab Current version only allows user to specify different # s of subjects Assumes # of runs for future study will be the same Assumes between subject variability is same across subjects Doesn t control for multiple comparisons

42 fmripower

43 2006: Hayasaka Whole brain Voxelwise Random Field Theory Obtain power or sample size map MATLAB toolbox hlps:// sourceforge.net/ projects/powermap/

44 2006: Hayasaka Whole brain Voxelwise FWE or sample size map MATLAB toolbox hlps:// sourceforge.net/ projects/powermap/

45 2006: Hayasaka Whole brain Voxelwise FWE Obtain power or sample size map MATLAB toolbox hlps:// sourceforge.net/ projects/powermap/

46 2006: Hayasaka Whole brain Voxelwise FWE Obtain power or sample size map MATLAB toolbox sourceforge.net/ projects/powermap/

47 2014: Posthoc Power by Durnez Whole brain FWER/FDR peaks or clusters Es3mate power for a given study mul3plicity works in our favor Use mixture of distribu3ons to es3mate propor3on of nonac3ve

48 2014: Posthoc Power by Durnez Whole brain FWER/FDR peaks or clusters Es3mate power for a given study mul3plicity works in our favor Use mixture of distribu3ons to es3mate propor3on of nonac3ve

49 2014: Posthoc Power by Durnez Whole brain FWER/FDR peaks or clusters Es3mate power for a given study mul3plicity works in our favor Use mixture of distribu3ons to es3mate propor3on of non-ac3ve

50 Why no posthoc power for ROI s? Power is a function of alpha If you rejected your null, post hoc power is always less than 50% See The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis Hoenig et al, Amer. Stat cutoff cutoff

51 Why no posthoc power for ROI s? Power is a function of alpha If you rejected your null, post hoc power is always less than 50% See The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis Hoenig et al, Amer. Stat cutoff

52 You still need to do something You need to have data! You need to have a region that you re interested in Based on findings of other studies is best Anatomical hypothesis are also quite good DO NOT simply take the region that was active for your task. This is biased.

53 What if you don t have data? Try to get as much information from similar papers as possible authors for raw ROI data Try to get mean and sd estimates off of plots in paper Durnez approaches only require a group statistics map! Check out data bases

54 That s it!