Accelerating GATE simulations

Transcription

1 GATE Simulations of Preclinical andclinical Scans in Emission Tomography, Transmission Tomography and Radiation Therapy Accelerating GATE simulations Parallel computing and GPU GATE Training, INSTN-Saclay, October 2015 Albertine Dubois IMNC CNRS Orsay, France Simon Stute SHFJ CEA Orsay, France Sebastien Jan SHFJ CEA Orsay, France

2 GATE Simulations of Preclinical andclinical Scans in Emission Tomography, Transmission Tomography and Radiation Therapy Running GATE using distributed architectures GATE Training 2014, Saclay, France 2

3 Introduction Parallel computing Simultaneous use of multiple computing ressources to achieve one computational problem Different usual approaches Multi-thread = shared memory (e.g. OpenMP, pthread) Multi-CPU = split memory (e.g. MPI) Intrinsic parallelism = cut the job into smaller jobs Monte Carlo method is most of the time intrinsically parallel 3

4 Monte Carlo parallelism Event splitting Assume events are independent to one another Works for radiotherapy applications Just split the job into pieces according to the number of events to be simulated and generate scripts for the batch queuing system Time splitting When events can interact between one another Works for imaging applications (pile-up, dead times, etc) Just split the job into pieces according to the total length of the simulation and generate scripts for the batch queuing system Finally merge the results from all pieces 4

5 GATE cluster tools GATE provides tools for convenient use with clusters Jobsplitter: to split the main simulation into pieces Filemerger: to merge the root outputs from all pieces Job splitter Batch queuing system Condor, OpenMosix, OpenPBS, Xgrid Job merger 5

6 Example using condor platform Easy to use tools with simple command lines Split the job (pieces are located in a.gate directory) gjs -numberofsplits 4 -clusterplatform condor condorscript condor.scpt macro.mac Submit the job condor_submit macro.submit Merge the results (works only for root files) gjm.gate/macro/macro.split Works well on multi-threaded desktop computers too 6

7 GATE Simulations of Preclinical andclinical Scans in Emission Tomography, Transmission Tomography and Radiation Therapy Running GATE using NVIDIA GPU GATE Training 2014, Saclay, France 7

8 Simulation strategy Using GPU needs specific language transcription Long, tedious, and needs thorough validation Move only high-cost parts of the simulation on GPU Hybrid simulation: GPU + CPU Current GATE strategy Phantom side on GPU One particule per thread CUDA kernels for specific physics and Monte Carlo navigation Detector side on CPU Phase space between phantom and detector Go back to GATE navigation and hits processing 8

9 GATE GPU modules Current GATE GPU modules Only for voxelized phantom (very time consuming) Only for PET and CT applications (validated in Bert et al 2013) GATE GATE-GPU GATE GATE-GPU GATE-GPU X 60 GATE GATE-GPU X 70 GATE 9

10 The PET module Only few commands related to the source change Example: /gate/source/addsource my_source GPUEmisTomo /gate/source/my_source/attachphantomto my_voxelized_phantom /gate/source/my_source/setgpubuffersize /gate/source/my_source/setgpudeviceid 1... Just declare your source as GPUEmisTomo Then attach the source to your voxelized phantom Set the size of phase space buffer (sets when CPU relieves GPU) Choose your GPU device (in case you have several) The rest is unchanged 10

11 The CT module Only few commands to add: Example: /gate/actor/addactor GPUTransTomoActor my_gpuactor /gate/actor/my_gpuactor/attachto my_voxelized_phantom /gate/actor/my_gpuactor/setgpubuffersize /gate/actor/my_gpuactor/setgpudeviceid 1... Just add a dedicated actor that do the link between the CPU and GPU Then attach the actor to your voxelized phantom Set the size of phase space buffer (sets when CPU relieves GPU) Choose your GPU device (in case you have several) The rest is unchanged 11