Op#miza#on of CUDA- based Monte Carlo Simula#on for Radia#on Therapy. GTC 2014 N. Henderson & K. Murakami

Transcription

1 Op#miza#on of CUDA- based Monte Carlo Simula#on for Radia#on Therapy GTC 2014 N. Henderson & K. Murakami

2 The collabora#on Special thanks to the CUDA Center of Excellence Program Makoto Asai, SLAC Joseph Perl, SLAC Andrea DoO, SLAC Koichi Murakami, KEK Takashi Sasaki, KEK Margot Gerritsen, ICME Nick Henderson, ICME 2

3 Radiotherapy simula#on methods Analy#c #me: seconds to minutes accurate within 3-5% used in treatment planning Monte Carlo #me: several hours to days of CPU #me accurate within 1-2% used to verify treatment plans in certain cases 3

4 Project overview GPU based dose calcula#on for radia#on therapy CT data Input Processing GPU/CUDA Dose calc. Output DICOM Visualiza#on gmocren Features GPU code based on Geant4 Voxelized geometry DICOM interface Material is water with variable density Limited Geant4 EM physics: electron/positron/gamma Scoring of dose in each voxel Process secondary par#cles

5 Geant4 High Energy Physics Space & Radia#on Medical Physics ATLAS LISA gmocren Images from: Geant4 gallery and gmocren

6 Basic idea Par#cles are transported through material in steps Physics processes act on par#cles and may generate secondaries Par#cles are removed when out of domain or out of energy

7 A single step

8 G4CU: CUDA implementa#on Each GPU thread processes a single par#cle un#l the par#cle exits the geometry Each thread has a stack for secondary par#cles Every thread takes a step in each itera#on Algorithm is periodically interrupted for various ac#ons Periodic actions Check termination conditions Generate primary particles Pop secondary particles Check resources Take step 8

9 G4CU: parallel execu#on kernel launches along-step post-step thread 0 thread 1 thread 2 gamma wait transport Compton scattering wait wait gamma wait transport wait Photoelectric effect wait electron ionization transport wait wait Bremsstrahlung CUDA kernels are applied to all par#cles Threads must wait if physics process does not apply to par#cle Parallel execu#on is achieved: maintenance opera#ons transport process same process on same par#cle

10 Performance and valida#on Hardware and so`ware: GPU: Tesla K20 Kepler 2496 cores, 706 MHz, 5GB GDDR5 (ECC) CPU Xeon X5680 (3.33GHz) single thread opera#on SDK: CUDA 5.5 Geant4 release 9.6 p2

11 Phantom geometry 30 cm 30.5 cm 10 cm water {water, lung, bone} water 100 cm 5 cm 20 cm voxel size: 5mm x 5mm x 2mm

12 Par#cle sources 20 MeV electron / 6 MeV photons (mono- energy) 6 MV & 18 MV spectrum photons

13 Depth dose distribu#on: water phantom 6MeV photon, 20 MeV electron dose along central axis Sep/25/ th Geant4 Collaboration meeting 18

14 Comparisons for slab phantoms 6MV photon, Lung/Bone as inner slab material

15 Visualiza#on with gmocren Image for demonstra#on purposes only! 50M 6MeV photons Pencil beam configura#on

16 Challenges Geant4 is complex Implement targeted subset Varied character of physics process Bad occupancy No clear op#miza#on target Random simula#on Thread divergence Lookup (interpola#on) tables Thread divergence Non- ideal memory access Energy deposi#on in global array Possible race condi#on Non- ideal memory access

17 Process Physics Processes Lines of code Ini#aliza#on kernel registers Step length kernel registers Ac#on kernel registers Compton scaiering Photo electric effect Gamma conversion Bremsstrahlung Ioniza#on (18,18) (60,36) Scaiering Positron annihila#on Electron removal Transport (20,22)

18 Physics Profile (6 MeV gamma) Sum of Time(%) step along-step after-step other init at-rest Grand Total em-helper ionization multiple-scattering management transport bremsstrahlung positron-annihilation compton-scattering gamma-conversion photo-electric-effect electron-removal Grand Total

19 Op#miza#on 1: dose storage Old idea: use thrust to join and reduce dose stacks into global array Beier idea: use atomicadd Speedup: ~1.5 (at the #me) index dose dose stack i join and sort dose stack j reduce by index

20 Op#miza#on 2: device configura#on 6 MeV gamma pencil beam 3,000,000 events Table shows run #me / shortest #me Voxels: 61 x 61 x = 22 seconds ~ 136 events / ms threads per block blocks threads per block blocks

21 Other op#miza#ons New hardware! Refactoring New features some#me slow us down Results from: 512 x 512 x 256 geometry 6 MeV gamma pencil beam Date Hardware Feature Time (min) Events / ms March C May K June K20 atomicadd July K20 mul#ple scaiering August K20 world volume Current K20 Refactoring, device config

22 Comparison to Geant4 on CPU Geometry:61 x 61 x 150 GPU: Tesla K20 CPU: Xeon X core, 12 thread Our measurement was with single threaded G4 New G4 is mul#- threaded We mul'ply by 12 to get CPU events / ms CPU GPU 20 MeV electron (pencil) 6 MeV photon (pencil) Time (h) Events / ms / core Events / ms Time (min) Events / ms speedup (GPU/core) speedup (GPU/CPU)

23 Other op#miza#on experiments 6 MeV gamma pencil beam 1,000,000 events Voxels: 61 x 61 x 150 Experiment Run #me (s) Events / ms standard prefer L1 cache disable L1 cache remove atomicadd limit to 32 registers no thread divergence, no atomic add no thread divergence, no atomic add, limit to 32 registers add bounds checking

24 Going forward Hope to get factor 1.5 speed up from spliong par#cles into separate CUDA streams Reduce thread divergence Allow concurrent kernel launches Some possible benefits from using texture memory and hardware interpola#on for lookup tables Look for other opportuni#es in expensive kernels

25 Acknowledgements NVidia for support of the CUDA Center of Excellence Koichi Murakami Makoto Asai, Joseph Pearl, Andrea SLAC Takashi KEK Akinori Kimura, Ashikaga Ins#tute of Technology 25