High Performance Computing in Neuroimaging using BROCCOLI. Anders Eklund, PhD Virginia Tech Carilion Research Institute

Transcription

1 High Performance Computing in Neuroimaging using BROCCOLI Anders Eklund, PhD Virginia Tech Carilion Research Institute

2 Outline What is BROCCOLI? What is high performance computing and GPUs? Why are GPUs needed in neuroimaging? GPUs in medical imaging, neuroscience, statistics, machine learning What can BROCCOLI do? Demos Future

3 What is BROCCOLI? BROCCOLI is a new software package for fmri analysis The main advantage of BROCCOLI is that it is much faster compared to other fmri softwares BROCCOLI performs all calculations in parallel and can, for example, run on CPUs, Nvidia GPUs and AMD GPUs

4 Is a new software package necessary? The importance of processing package development has been underrated. SPM, FSL, AFNI all fundamentally push the field as they are distributed. The field progress is directly related to how dynamic and cutting edge these packages are since we all rely on them. Peter Bandettini, NSF: Neuroimaging Final Workshop, May 23, 2014

5 What is high performance computing? Aggregate computing power to obtain much higher performance, than possible with a regular computer Used to solve large problems in science & engineering PC clusters (many computers) GPUs (graphics cards)

6 Graphics card What is a GPU? Graphics processing unit The part of a graphics card that performs calculations Calculations are done in parallel, for example to render a large number of pixels at interactive frame rates GPU

7 A poor man s supercomputer (~3000 USD)

8 Giga floating point operations per second (GFLOPS) CPU vs GPU Theoretical Performance 8 From the Nvidia CUDA Programming Guide

9 Serial vs parallel computing Adding two vectors, N elements Serial computing, one thread performs all operations, one at a time (C) Parallel computing, N threads, run in parallel (CUDA) for (int i = 0; i < N; i++) { } C[i] = A[i] + B[i]; int i = blockidx.x * blockdim.x + threadidx.x; if (i < N) C[i] = A[i] + B[i];

10 Why are GPUs needed in neuroimaging? Higher temporal and spatial resolution More subjects Demanding image processing algorithms Demanding statistical algorithms Demanding machine learning algorithms

11 Why are GPUs needed in neuroimaging? Higher temporal and spatial resolution Standard resolution dataset (4 x 4 x 4 mm, 0.5 Hz), 64 x 64 x 30 x 430 High resolution dataset (2 x 2 x 2 mm, 1.4 Hz), 104 x 90 x 72 x times more data!

12 Why are GPUs needed in neuroimaging? More subjects Data sharing initiatives enable large scale analyses OpenfMRI.org, 420 subjects 1000 Functional Connectomes Project, 1200 subjects Human Connectome Project, goal 1200 subjects

13 Why are GPUs needed in neuroimaging? Demanding image processing algorithms Image registration, to align two images or volumes Image segmentation, to extract a certain part of a dataset Image denoising, to suppress noise and improve image quality

14 Why are GPUs needed in neuroimaging? Demanding statistical algorithms Non-parametric statistics (e.g. permutation tests, bootstrap, jackknife) Cross validation Markov Chain Monte Carlo (MCMC) simulation Dynamic causal modeling

15 Why are GPUs needed in neuroimaging? Demanding machine learning algorithms Support vector machines Search light, a classifier in each voxel Independent component analysis Deep learning Bayesian methods

16 Why are GPUs needed in neuroimaging? Human connectome project; 1200 subjects, 3 GB of resting state data per run, 4 runs Apply a permutation test (10,000 permutations) to each subject and each run Equivalent to analyze 144 petabytes of data (1200 * 3 GB * 4 * = GB)

17 How are GPUs being used in fields related to neuroimaging?

18 GPUs in Medical Imaging, Modalities Eklund et al., Medical Image Processing on the GPU Past, present and Future, Medical Image Analysis, 2013

19 GPUs in Medical Imaging, Algorithms Eklund et al., Medical Image Processing on the GPU Past, present and Future, Medical Image Analysis, 2013

20 GPUs in Neuroscience Neural field models, nonlinear integral differential equations Input Stimulus Simulating stochastic networks Estimating network probability distributions Output Brain Activity Baladron et al., Three Applications of GPU Computing in Neuroscience, Computing in Science & Engineering, 2012

21 GPUs in Statistics Non-linear regression Resampling based methods (bootstrap, permutation, jackknife, cross-validation) Markov Chain Monte Carlo (MCMC) simulation Random number generation directly on the GPU Suchard et al., Understanding GPU programming for statistical computation: Studies in massively parallel massive mixtures, Journal of Computational and Graphical Statistics, 2010

22 The Google Brain Project Deep learning using 1.7 billion parameters 10 million YouTube video thumbnails, 200 x 200 pixels No labels, unsupervised learning 16,000 CPU cores (1000 machines), 3 days of processing Recognizes human faces and cat faces Le, Q., Building high-level features using large scale unsupervised learning, ICASSP, 2013

23 GPUs in Machine Learning Used 3 GPU machines instead of ,432 GPU cores $30,000 instead of $5,000,000 4 kwatts instead of 600 kwatts Same processing time Coates et al., Deep learning with COTS HPC Systems, JMLR, 2013

24 GPUs vs Supercomputers GPUs are much cheaper GPUs are much more power efficient GPUs have a much smaller physical size GPUs enable local data storage Do not have to share the resources with anyone else

25 Why are GPUs uncommon in neuroimaging?

26 Software packages for fmri analysis No software package with GPU support for fmri analysis Existing softwares are not prepared for the medical data explosion and demanding algorithms BROCCOLI is intended to fill this gap

27 How can a GPU be programmed?

28 CUDA CUDA = Compute Unified Device Architecture Released in 2007, based on the C programming language Several highly optimized libraries (e.g. CUFFT, CURAND, CUBLAS, CUSPARSE) Only works for Nvidia GPUs 28

29 OpenCL OpenCL = Open Computing Language Works for many types of hardware CPUs, GPUs, field programmable gate arrays (FPGAs), digital signal processors (DSPs), Intel Xeon Phi, Not as many libraries as CUDA (yet) 29

30 BROCCOLI uses OpenCL Can use both Nvidia and AMD GPUs Can also run the same code on a CPU Not necessary to buy a GPU to try BROCCOLI!

31 BROCCOLI library Ongoing Future

32 BROCCOLI bash functions GetOpenCLInfo, lists all OpenCL platforms and devices RegisterTwoVolumes, linear and non-linear registration FirstLevelAnalysis, including permutation testing and MCMC RandomiseGroupLevel, permutation testing MotionCorrection

33 Demo 1, OpenCL platforms and devices

34 Image registration in BROCCOLI Most image registration algorithms are based on optimizing an intensity based similarity measure (correlation, mutual information, ) BROCCOLI instead relies on the concept of local phase from quadrature filters

35 Quadrature filters Spatial domain Line detector (even) Real part Edge detector (odd) Imaginary part Knutsson, H., What s so good about quadrature filters? ICIP, 2003

36 Quadrature filters Frequency domain Knutsson, H., What s so good about quadrature filters? ICIP, 2003

37 Local phase

38 Minimizing the optical flow equation The linear registration algorithm in BROCCOLI is based on minimizing the optical flow equation The equation cannot be solved, since there is one equation and three motion parameters per voxel

39 Minimizing the optical flow equation Use a parametric model of the motion field

40 Minimizing the optical flow equation Put the parametric model into the error expression Calculate the derivative, with respect to p

41 Minimizing the optical flow equation Set the derivative to 0, to find the minimum Solve the equation system to get the best parameters, no optimization algorithm required

42 Phase based optical flow Replace intensity difference with phase difference Replace image gradient with phase gradient Add a certainty for each voxel (e.g. the magnitude of the quadrature filter response)

43 Linear registration algorithm Apply 3 quadrature filters (along x, y and z) Calculate phase differences, phase gradients and certainties in each voxel Setup a linear equation system for the entire image Solve the equation system to get the best transformation parameters Apply transformation parameters using interpolation Iterate

44 Non-linear registration The non-linear registration algorithm used in BROCCOLI is called the Morphon The Morphon also uses the concept of local phase Solve one equation system in each voxel, instead of one equation system for the entire image Knutsson & Andersson, Morphons: Segmentation Using Elastic Canvas and Paint on Priors, ICIP, 2005

45 The Morphon L2 norm minimization in each voxel Structure tensor Filter direction Certainty Phase difference Displacement vector

46 The local structure tensor T Describes the strength and the orientation of the signal in an image or a volume Analogous to the tensor in diffusion tensor imaging Apply filters along different directions (at least 6 in 3D) and combine the filter responses to a tensor Knutsson, H., Representing local structure using tensors, SCIA, 1989

47 The Morphon Calculate the derivative with respect to d, set it to 0, solve the equation system, no optimization algorithm required

48 Scale pyramid Find approximate solution at a low resolution Improve the solution by moving to finer and finer scales Downsampling a factor 8, 4, 2, 1

49 RegisterTwoVolumes - Bash RegisterTwoVolumes inputvolume.nii referencevolume.nii [options] Some of the options are -platform -device -lowestscale OpenCL platform to use OpenCL device to use Highest downsampling factor

50 Demo 2, linear and non-linear registration in BROCCOLI

51 Demo 2, Summary Can easily change the used platform Linear and non-linear registration to a 2 mm MNI template in 1 second Linear and non-linear registration to a 1 mm MNI template in 5 seconds The registration is rather quick even with a CPU

52 Demo 2, Summary Non-linear registration of 200 T1 volumes to a brain template (182 x 218 x 182) FSL FSL, 4 terminals AFNI AFNI OpenMP BROCCOLI Intel CPU BROCCOLI Nvidia GPU BROCCOLI AMD GPU 116 hours 29 hours 110 hours 32 hours 3.5 hours 15 minutes 20 minutes

53 Average similarity metrics from 200 registrations Correlation Mutual information Sum of squared differences

54 First level analysis in BROCCOLI Registration of T1 volume to a brain template (using linear and non-linear registration) Registration of one fmri volume to the T1 volume Motion correction Smoothing Statistical analysis

55 FirstLevelAnalysis - Bash FirstLevelAnalysis fmri.nii T1.nii braintemplate.nii regressors.txt contrasts.txt [options] Some of the options are -platform -device -permute -bayesian OpenCL platform to use OpenCL device to use Apply permutation test Run MCMC simulation

56 Demo 3, first level analysis in BROCCOLI

57 Demo 3, Summary First level analysis in ~10 seconds, for a 2 mm MNI template First level analysis in ~25 seconds, for a 1 mm MNI template

58 Demo 3, Summary Intermediate results take up a lot of disk space Original dataset 500 MB, motion corrected 500 MB, smoothed 500 MB, Unnecessary to store all intermediate results if BROCCOLI is used, recalculate them if necessary

59 How should the statistical maps be thresholded? How should we correct for multiple testing?

60 Parametric vs non-parametric statistics Most fmri software packages rely on parametric statistical algorithms to calculate p-values The parametric algorithms are based on a number of assumptions about the data These assumptions become more critical when we correct for multiple testing

61 Eklund et al., Does parametric fmri analysis with SPM yield valid results? An empirical study of 1484 rest datasets, NeuroImage, 2012

62 Parametric vs non-parametric statistics Non-parametric algorithms (e.g. a permutation test) can sometimes provide more reliable results Empirically estimate the null distribution, by analyzing data similar to the original dataset Obtain similar data by randomly permuting volumes

63 Eklund et al., Does parametric fmri analysis with SPM yield valid results? An empirical study of 1484 rest datasets, NeuroImage, 2012

64 Eklund et al., Does parametric fmri analysis with SPM yield valid results? An empirical study of 1484 rest datasets, NeuroImage, 2012

65 Demo 4, permutation based first level analysis in BROCCOLI

66 Demo 4, Summary Can easily add a permutation test by using the option -permute Gives permutation based p-values, corrected for multiple comparisons 1,000 permutations in about 6 seconds

67 What about Bayesian statistics?

68 Bayesian fmri analysis Most Bayesian fmri analyses rely on variational Bayes to obtain an analytical expression of the posterior distribution Variational Bayes is based on a number of assumptions and approximations For this reason, BROCCOLI instead uses MCMC, runs many MCMC chains in parallel Ferreira da Silva., cudabayesreg: Parallel implementation of a Bayesian multilevel model for fmri data analysis, Journal of statistical software, 44, 2011

69 Demo 5, Bayesian first level analysis in BROCCOLI

70 Demo 5, Summary Can easily do MCMC based Bayesian analysis by using the option -bayesian Currently only supports 2 regressors and AR(1) Currently only supports first level analysis 1,000 MCMC iterations in about 20 seconds (straight forward Matlab implementation: 2.5 h)

71 Permutation based second level analysis in BROCCOLI Very similar to randomise in FSL Supports voxel level, cluster extent and cluster mass inference Threshold free cluster enhancement being implemented

72 RandomiseGroupLevel - Bash RandomiseGroupLevel volumes.nii [options] Some of the options are -platform -device -groupmean -design -contrasts -inferencemode -mask OpenCL platform to use OpenCL device to use One sample t-test Regressors Contrasts Voxel, cluster extent, cluster mass Restrict analysis to mask

73 Demo 6, permutation based second level analysis in BROCCOLI

74 Demo 6, Summary Usage very similar to randomise in FSL BROCCOLI gives the same results as randomise for one regressor, not yet for several regressors BROCCOLI is ~85 times faster for 1 regressor, ~250 times faster for 25 regressors Lower speedup when compared to the parallel version of randomise

75 What about the future?

76 Future extensions of BROCCOLI Permutation based group level analysis using a method similar to FLAME in FSL or 3dMEMA in AFNI (includes the variance of each subject)

77 Future extensions of BROCCOLI Statistical analysis using canonical correlation analysis (CCA) Combines neighbouring voxels or filter responses in an optimal way, can give a higher statistical power Use permutation test to estimate complicated null distribution Friman et al., Adaptive analysis of fmri data, NeuroImage, 2003

78 Future extensions of BROCCOLI Support vector machines (SVM) Permutation based SVM, to get p-values of the classifier weights SVM relies on matrix operations, CUBLAS for CUDA, clblas for OpenCL Catanzaro et al., Fast support vector machine training and classification on graphics processors, ICML, , 2008

79 Future extensions of BROCCOLI Independent component analysis (ICA) Permutation based ICA, to get p-values of time series and spatial maps ICA can be performed using matrix operations, CUBLAS for CUDA, clblas for OpenCL Raimondo et al., CUDAICA: GPU optimization of infomax-ica EEG Analysis, Computational Intelligence and Neuroscience, 2012

80 Thanks to Stephen LaConte Postdoc supervisor Paul Dufort GPU guru Mattias Villani Bayesian guru Daniel Forsberg Image registration guru Cyrus Eierud Beta tester Morgan Hough Python guru Satrajit Ghosh Python guru Miha Cancula Python guru Hans Knutsson Phd supervisor Mats Andersson Phd supervisor

81 Questions or comments? Beta testers needed!