CUDA- based Geant4 Monte Carlo Simula8on for Radia8on Therapy. N. Henderson & K. Murakami GTC 2013

Transcription

1 CUDA- based Geant4 Monte Carlo Simula8on for Radia8on Therapy N. Henderson & K. Murakami GTC

2 The collabora8on Makoto Asai, SLAC Joseph Perl, SLAC Koichi Murakami, KEK- SLAC Takashi Sasaki, KEK Margot Gerritsen, ICME Nick Henderson, ICME Special thanks to the CUDA Center of Excellence Program 2

3 20,000,000 radiotherapy treatments per year in US 3

4 Image: capitalmedical.com 4

5 liver kidney Image: James M. Balter and Marc L. Kessler, JCO March 10, 2007 vol. 25 no

6 Simula8on methods Analy8c 8me: seconds to minutes accurate within 3-5% used in treatment planning Monte Carlo 8me: several hours to days of CPU 8me accurate within 1-2% used to verify treatment plans in certain cases 6

7 Geant4 Toolkit Enables Monte Carlo simula8on of par8cles travelling through and interac8ng with mayer Allows modeling of complex geometries Covers all elementary par8cles and nuclei for a wide energy range 7

8 Geant4 Applica8ons High Energy Physics Space & Radia8on Medical Physics ATLAS LISA gmocren Images from: Geant4 gallery and gmocren 8

9 Geant4 101 Geant4 simulates par8cles travelling through and interac8ng with mayer Example: photoelectric effect Image: CCSA by Wolfmankurd on Wikipedia 9

10 Paralleliza8on challenges in Geant4 Large and complex code base Sophis8cated geometry framework Elaborate physics models Branching, look- up tables, single- thread op8miza8ons 10

11 Thank goodness The simula8on is embarrassingly parallel! (the par8cles are independent) 11

12 Requirements for X- ray Radiotherapy Geometry is a voxelized box Physics is limited to low- energy electromagne8cs Material is modeled as water with different densi8es 12

13 Low energy electromagne8cs Gamma Compton scayering Photoelectric effect Gamma conversion Electron/positron Ioniza8on Bremsstrahlung Positron annihila8on Todo: Mul8ple scayering Image: CCSA by Man8corp on Wikipedia Image: public domain 13

14 What makes a physics process? Sample for angular and energy distribu8ons of secondary par8cles Deposit energy to material Produce secondary par8cles that must be tracked at a later point 14

15 Tracking algorithm Par8cles are tracked through space Each discrete move is called a step Physics process may occur along step, ager the step, or both step 1 par8cle secondary par8cle step 2 15

16 How are processes selected? Each process has an interac8on length (dx) Process with shortest dx is selected Ager the step dx is decreased or resampled 16

17 Energy deposi8on Happens along the step for the ioniza8on process May happen at the end of the step Secondary par8cles with too low energy are not generated, but treated as point- like energy deposi8ons step 1 par8cle secondary par8cle x step 2 17

18 G4CU: CUDA- base MC for RT Important data structures Algorithm summary Details Parallel stack Look- up tables 18

19 Struct- of- arrays data payern struct ParticleArray { // length of arrays int length; // kind of particle ParticleKind *kind; // position float *x, *y, *z; // direction float *dx, *dy, *dz; // particle energy float *energy; // voxel index int *vx, *vy, *vz, *vid; }; Common payern in CUDA to allow for coalesced memory access Experiments with transport showed this to be 3-4x faster than AOS 19

20 G4CU: algorithm query all processes to select step size reduce dose stacks to global array main loop apply all continuous processes decrease interaction lengths pop secondaries from stacks generate primary particles management apply limiting discrete process check termination conditions 20

21 Some details A process does a few things: 1. Changes direc8on and momentum of primary par8cle 2. Generates secondary par8cles 3. Deposits energy to the material We use thread local stacks to handle 2 and 3 21

22 Parallel stacks start end data pos template <typename T> struct pstack { // number of stacks int stack_num; // size of each stack int stack_size; // starts and ends of stacks int *start, *end; // stack positions int *pos; // stack data array T *data; }; 22

23 Dose reduc8on & storage Energy dose is stored in a thread local variable When par8cle moves to new voxel, dose is pushed onto stack Dose reduc8on is performed periodically We use Thrust index dose dose stack i join and sort dose stack j reduce by index

24 Used for interac8on length computa8on 40 bins, log spaced Linear and spline interpola8on Bremsstrahlung also uses 2D interpola8on Look- up tables 24

25 Benchmarks Configura8on: Geometry: 512 x 512 x 256 voxels Dose reduc8on frequency: every 200 itera8ons 128 blocks with 256 threads Primary par8cle is a 6 MeV gamma Tesla C

26 Simula8on 8me 100 million primary par8cles Time: 72 minutes ~ 23.1 primary par8cles per ms ~ 50-60x speedup over Geant4 on 1 CPU 26

27 Profile Component Percentage of overall 1me Physics processes 50 Energy dose reduc8on 30 Interac8on length 18 Run management 2 27

28 Physics process breakdown Process Par1cle Percentage of overall 1me Bremsstrahlung e- / e+ 23 Pair produc8on γ 7.5 Transport all 7 Photoelectric effect γ 7 Ioniza8on e- / e+ 3 Compton scayering γ 1.5 Positron annihila8on e+ 1 28

29 Dose distribu8on 29

30 Acknowledgements Geant4 Collabora8on, see Geant4 a simula1on toolkit, Nuclear Instruments and Methods in Physics Research A 506 (2003) Geant4 developments and Applica1ons, IEEE TransacGons on Nuclear Science 53 No. 1 (2006) NVIDIA & CCOE Program People: Koichi Murakami, KEK Makoto Asai, SLAC Joseph Perl, SLAC Takashi Sasaki, KEK Margot Gerritsen, ICME 30