GEANT4 particle simulations in support of the neutron interrogation project

Transcription

1 Defence Research and Development Canada Recherche et développement pour la défense Canada GEANT4 particle simulations in support of the neutron interrogation project ANS Technologies Inc. Contract Scientific Authority: A.A. Faust, DRDC Suffield The scientific or technical validity of this Contract Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of Defence R&D Canada. Defence R&D Canada Contract Report DRDC Suffield CR December 2007

2

3 GEANT4 particle simulations in support of the neutron interrogation project ANS Technologies Inc. University of Montreal, Lab. Rene-J.A.-Levesque P.O. Box 6128, Station CV Montreal QC H3C 3J7 Contract Number: W R108/001/EDM Contract Scientific Authority: A.A. Faust ( ) The scientific or technical validity of this Contract Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of Defence R&D Canada. Defence R&D Canada Suffield Contract Report DRDC Suffield CR December 2007

4 Her Majesty the Queen as represented by the Minister of National Defence, 2007 Sa majesté la reine, représentée par le ministre de la Défense nationale, 2007

5 Geant4 particle simulation in support of the neutron interrogation project Progress Report, by ANS Technologies Inc. for Dr. Anthony Faust, Threat Detection Group, Suffield project anst-c Objective In this report, we provide an explanation and propose a solution to the event limit problem encountered during the previous report. The data persistency were included using Reflex with ROOT I/O method; the analysis tools was written for extracting a ROOT readable file from the Reflex/ROOT format to an independent ROOT ntuple file; the pile-up analysis tools are also provided. In addition, we have run a full simulation with a reasonable event number for testing the new cluster configuration. 1 Introduction We have received a 64-bit quad-core with 8G of memory Supermicro server from Threat Detection Group. The final setup of this server will be configured as a head node. In our facility, we have a 2G dual-core Dell server and it can be used as a head or worker node depending the needs. Originally, in our understanding, the plan was to setup a Beowulf type of cluster using clustermatic distribution (BProc). This cluster can run 32 or 64-bit OS depending on which OS is better for TNA simulations. Since we already knew how the TNA simulations run on a 32-bit cluster, it was worth testing it on a 64-bit cluster to see if it can boost the simulations performances. So a test Beowulf cluster using 64-bit OS with our 2G Dell server as head node was setup and the TNA simulation programs were recompiled for 64-bit running Scientific Linux CERN 4 (SLC4). The test seems to work fine, however the memory used increases very rapidly. This behavior has an explanation which we will discuss in details later in the report. We later learned that DND had changed their distribution to a Rocks distribution. In order to re-adapt to this new situation, a test Rocks cluster has been configured using 64-bit Centos 4 OS. The Rocks cluster does not require the BProc option to be compiled into the OpenMPI any longer. The communication between the head node and the worker node uses the ssh protocol. In this case, we need to reconfigure TOP-C and the other related external softwares to be able to run the TNA simulation on the Rocks cluster. OpenMPI is used as the default MPI and will be updated regularly. In previous reports we worked with geant4.8.1.p01 and since that time another version of geant4 was released. The latest version, at the moment of writing of this report, is geant4.9.0.p01. In this version, there are several changes that affect the ParTNAmain, for instance the Hadron physics has been moved and the neutron data file is new. In principle, we can run a 32-bit application as well as a 64-bit application on a 64-bit OS. We need to compile all the external packages and ParTNAmain in 32-bit. For this task, it is not 1

6 always easy to put the right flags at the right places. The persistency using ROOT/IO have been included in ParTNAmain (parallel version of the TNA simulation) as well as in the TNAmain (serial version). The pulse pile-up analysis of the simulated energy deposition in the TNA detector will be presented. We will also present the results of a simulation for 10 million events which is 100 times more than in the last report. 2 Clusters setup In this section, we present a short description of the clusters setup for a 64-bit Beowulf cluster with BProc and a 64-bit Rocks cluster using Centos Beowulf cluster (BProc) We had already reported a working 64-bit Beowulf cluster in the previous report. In this previous report, we had problems compiling CPPGDML using the gcc4.x series compiler. We found that this problem can be solved using the gcc3.4 series compiler. The OS kernel version is 2.6.9, which is similar to the version in CentOS 4.5. We note that the Supermicro s ethernet controller is an Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper), the default driver from the kernel did not work at all. We found that the e worked fine. In order to test the Supermicro server as worker node, we need to compile and incorporate the ethernet driver as modules in the BProc kernel and than load it as stage 2 s modules. The easy method is to compile the ethernet driver module and to copy it into the appropriate kernel library directory and then to produce a second stage boot image and initrd using the following command line: beoboot -2 -n -i -o stage2 Once this is done, one can load the image from the worker node using PXEboot. It is also possible to load the ethernet driver module by modifying the original clustermatic distribution and then booting the worker node using the boot CD. The jobs submission using mpirun worked as before, there were no modifications needed for the submission scripts. However, we noted that the master process memory utilization increased very rapidly. This can be explained by the fact the 64-bit program allocated more memory that the 32-bit version and ran much faster than 32-bit. This is why in the same amount of simulation time the memory needs increase faster compared to the 32-bit OS simulation. More details of memory utilization of the ParTNAmain will be given in the Rocks cluster section. 2.2 Rocks Cluster By following the installation guide for Rocks 4.3 distribution from the Rocks webpage [1], the installation Rocks cluster was simple. However, several parameters related to the network were 2

7 not configured correctly and a manual intervention was required after the installation. In Rocks 4.3, we have a choice of operating systems, we understood DND preferred to use the CentOS (in this case, the OS version is 4.5). In this setup, we used the Supermicro server as the head node and the Dell server as worker node. Later, when we run TNA simulations on the Rocks cluster we discovered that there were intermittent periods of network interruptions from the front ethernet card (eth0). We first thought it was an IP conflict since the internal ethernet card (eth1) was working. After debugging, we found that there was no IP conflict causing this problem but instead the problem came from the ethernet driver. A new driver was installed as a module in Rocks 4.3 in order to have a stable network communication between the nodes. Since that time we have experienced only two to three network disconnections under heavy load. Since the communication protocol between the head node and worker node is ssh, we have to reconfigure OpenMPI and TOPC, and recompile ParTNAmain. For the Rocks cluster configuration, we had a hard time to figure out how to make mpirun work. It seems that not all the local environment variables are exported to the worker node as it is done in the case of the Beowulf cluster. In particular, we needed to force the following variables to the mpirun (orterun) as options: #geant4.8.1 mpirun --prefix $MPI_DIR -np 3 -machinefile /home/chen/machines-a \ -x G4LEVELGAMMADATA -x G4RADIOACTIVEDATA -x G4RADIOACTIVEDATA -x G4LEDATA \ -x G4ELASTICDATA -x NeutronHPCrossSections -x ROOTSYS \ /home/chen/geant4/bin/linux-g++/partnamain $TOPC_opt1./$macroFile >output1.out We note that the prefix option was necessary as well as the -x option and it had to be exported individually. For geant4.9.0.p1, we needed to replace NeutronHPCrossSections by G4NEUTRONHPDATA. In addition, the LD LIBRARY PATH cannot be passed by the user, we have to set it in /etc/ld.so.conf.d/pargeant4.conf as root user. This limitation is very inconvenient for the user. Maybe later we can find a way to overcome this limitation. Inside the pargeant4 file, it looks like (for 64-bit): /opt/software64/clhep/lib /opt/software64/xerces-c-src_2_7_0/lib /opt/software64/aidajni-3.2.3/lib/linux-g++ /opt/software64/jdk1.6.0_02/jre/lib/amd64/server /opt/software64/root/lib #geant4.8.1 /opt/software64/cppgdml/build/linux-g++/lib /opt/software64/geant4.8.1.p01/lib/linux-g++ #geant4.9.0 /opt/software64/cppgdml-g4.9.0/build/linux-g++/lib 3

8 /opt/software64/geant4.9.0.p01/lib/linux-g++ and for 32-bit /opt/software32/clhep/lib /opt/software32/cppgdml/build/linux-g++/lib /opt/software32/aidajni-3.2.3/lib/linux-g++ /opt/software32/jdk1.6.0_03/jre/lib/i386/server /opt/software32/root/lib /opt/software32/geant4.9.0.p01/lib/linux-g++ /opt/software32/python-2.4.4/lib /opt/software32/root/lib In a Rocks cluster, we need to tell the mpirun which machine to use and also how many slots are available by using the -machinefile option. The option file looks like: ansq1.local slots=1 compute-0-0.local slots=2 This will launch one master process in ansq1.local and two slave processes in compute-0-0.local. We can also run the slave processes in the head node by setting the machinefile as: ansq1.local slots=3 It is not recommended to run slave processes in the master node on a Beowulf cluster since the master control the slaves and the slaves can be rebooted. This is what we want to avoid in a running Beowulf cluster. Overall for the 64-bit Rocks cluster, we have been able to compile TNAmain and ParTNAmain in 64-bit and 32-bit. For 32-bit, we had to force the compiler in 32-bit with the option -m32. The main problem was to put this option at the right place. We have also experienced problems with AIDA when an older java version was used. The working version is jdk and higher. 3 Persistency The persistency was included as a module in RootIO.cc using the lcgdict tool to create the Reflex dictionary for geant4 classes and use Root/IO to save as ROOT files. The Root/IO also has the ability to read Root persistency files during the simulations. In our case, we will produce files for the TNA detector, the TNT and Soil separately because the soil (defined as sensitive detector to neutron) root file could become huge for large events simulations. The selection file for producing the dictionary looks like: 4

9 <lcgdict> <class name="std::basic_string<char>" /> <class name="g4string"/> <class name="g4vhit"/> <class name="tnatnthit"/> <class name="std::vector<tnatnthit*>"/> <class name="tnasoilhit"/> <class name="std::vector<tnasoilhit*>"/> <class name="tnadetectorhit"/> <class name="std::vector<tnadetectorhit*>"/> <class name="clhep::hep3vector"/> </lcgdict> The dictionary library is created by simply typing make dictionary This will produce a library file libclassesdict.so and an executable file readtnahits. The executable file allows one to extract and create ntuple files readable by Root. The libclasses- Dict.so is required by readtnahits to be able to extract Root persistency data to produce a simple Root ntuples file. 4 Data Collection and Simulations Results 4.1 Limitation First, let us to clarify an issue from the last report, we had reported that we had successfully run 60 million events on Gimili and also reported the results for events, saying that we would report for 60 millions later. The Gimili calculation included only the sensitive TNA detector. Adding the TNT and the soil as neutron sensitive detectors and storing ntuples will change considerablebly the memory used in the simulation. In the following section we try to understand memory utilization by TNAmain and ParTNAmain. This can help us to solve the event-limit problem and how to handle the future data collection when we will want to keep more data for post simulation analysis. In pargeant4, there is a memory allocation for the random seeds to insure the slaves use different seeds in order to have independent Monte-Carlo events. The global seeds allocation is done by using: // Setup random seeds for each slave g_seeds = (long*)calloc(n_event, sizeof(long)); where n event is the number event N from beamon(n). Naturally, this kind of allocation has 5

10 an upper limit on the memory utilization, which the test, originally, omitted. The correct prevention can be written as: // Checking g_seeds if out of memory exit; if (g_seeds == NULL ) { G4cout<<"g_Seeds: out of memory\n"<<g4endl; exit (1); } In 64-bit, long* is a 8 bytes segment and in 32-bit, long* is a 4 bytes segment. For example, when N=100 millions, ParTNAmain has a footprint of 900M for the master node and 891M for the slave nodes after the geometry has been loaded. When N=10 millions, ParTNAmain has a footprint of 227M for the master and 207M for the slaves after the geometry has been loaded. This illustrates how N affects the footprint. When the simulation started to run events, ParTNAmain started to collect data via the sensitive detectors; at the moment we have three Sensitive detectors: the TNADetectorSD, TNATntSD and TNASoilSD and also the AIDA histogram and ntuple hold data in the memory until the simulation program finishes then it frees the memory. We are not aware of any option available in AIDA that allows one to empty the memory to the file continuously as it does for the ROOT/IO persistency. The global memory utilization for storing simulated data is huge, sometimes it can reach 1.5Gb to 2Gb only for the soil sensitive detector for 100 million events. We have also tried hunting for memory leaks by using Valgrind which includes memcheck and massif. Running ParTNAmain in debugging mode using Valgrind did not allow us to find any memory leak. There is another important factor that contributes to the memory increase is the TOPC/Marsharling, this involves copying data forward and backward between the slaves and the master. To isolate the problem, we nead to run the TNA simulation in serial mode (run TNAmain). In this case, the TOPC part is removed and the footprint of the TNAmain at N=100 millions is 143M which is much smaller than the ParTNAmain. The rate of memory occupation increased from 77M/(million of events) for TNAmain to 200M/(million of events) for ParTNAmain. To simulate 100 million events, TNAmain needs 7.8G of memory and 21G of memory for ParTNAmain. The memory could be a virtual memory (swap). To solve this problem, the easy solution is to increase the memory to 32G for the head node and set a large swap partition. We can also divide the simulation into several simulations with different seeds each and add up the results at the end of the simulation. 4.2 Optimization and Performance According to our observation, the performance of TNA simulations has been enhanced by running a Rocks cluster. We note that in the Beowulf cluster the slave processes could not run at full capacity, contrary to the Rocks cluster, where all processes run at almost 99% all the time. We present below a measure of the simulation time needed for calculating one event per node (s/event): ParTNAmain: Gimili: (used 9 nodes as slaves) Rocks: (two slaves run in Supermicro (master)) 6

11 Rocks: (two slaves run in Dell (pentium D 3.0GHz)) TNAmain: Rocks: (one process on master node) The supermicro server is 15 times (if the calculation is based on a single node) as fast as gimili and 2.5 times as fast as the Dell server. 4.3 Simulation We present an update of the previous parallel Geant4 simulation of the full TNA detector geometry for 10 millions events using ParTNAmain for the energy deposition with geant4.8.1.p01 and geant4.9.0.p01. The upgrade from p0 to p01 clearly presented a problem. As we mentionned earlier there is a new neutron data set and the hadron physics list has been moved. This problem needs to be solved if we wish to use the latest version of geant4. We note that there is yet a newer version of geant4, released during the writing of this report, which has a new neutron data set; this could be the first thing to try. All the results below are simulated using geant4.8.1.p01 since we are having problems with geant4.9.0.p01. The neutron flux distribution for the TNA detector (histogram 20), soil (histogram 21) and TNT (histogram 22) is also updated in figure 2. The TNT is positioned at 5cm in the z-direction. Fig. 3 shows an update of the neutron fluence for the soil (left figure) and the TNT (right figure) as function of the distance defined in the previous report. Fig. 4 shows an update of the event-time ditribution of the photons hitting the sensitive detectors. Fig. 5 shows the neutron distribution in the soil as XYZ plotted with Root. With OpenGL we can have different viewpoints of the neutron distribution, as shown in Fig. 6. Fig. 7 shows the neutron flux at different surfaces Pile-up The pulse pile-up calculation is done using the Reflex ROOT/IO format file which stores the time, t, and the deposited energy, E d, of an event hitting the TNA detector. For the pile-up treatement algothrim, we follow Palomba et al. [2] s work. We first need to define the pulse rise time in the pre-amplier as τ R and time width in the shaped output pulse as τ P. Assuming two events with t 1 and t 2 hitting the TNA detector. If these two events occured at the atmost same time then there is a pile-up occurence; if t 2 t 1 < t R, the pulse shape is indistinguishable by any filter and we accept the pile-up and add the two event energies and record it. If the pile-up occurs between τ R t 2 t 1 < τ P, the time difference is large and the rejection filter can detect the pile-up, both events are rejected and no particle is stored. If τ P < t 2 t 1, the pulses are distinguishable by the filter, there is no pile-up occurence and both particles are stored. Fig. 8 shows the pile-up analysis using the algorithm described above with τ r = 100 ns and τ P = 3 µs. The red histogram represents the original energy deposition distribution and the 7

12 Figure 1: Energy deposition calculated using ParTNAmain for the full TNA detector geometry (Blue with geant4.8.1.p01 and red with geant4.9.0.p01). The TNT is placed at 5 cm in the z-direction. green one includes the calculation of the pile-up. 5 Conclusion A 64-bit Beowulf cluster and a 64-bit Rocks cluster were built and this allowed as to further test the TNA simulations. In these clusters, both 32-bit and 64-bit TAN simulations can be performed. The Rocks cluster seems to be much faster than the Beowulf cluster. A detailed analysis of the event limit problem has been done and by adding more memory we can help to solve the problem. We have also included the Root persistency and the pile-up analysis, as well provided new updated simulations of the TNA detectors. We also found that geant4.9.0.p01 presents some problems which need to be solved, unless we update to the newest version of the geant4. 8

13 Figure 2: Neutron flux (in mm 2 ) as a function of energy for the dectector (20), soil (21) and TNT (22). Appendix Problems encounted: ulimit -a The stack size is set at 10240, this should set to unlimited when run 32-bit application on 64-bit cluster. Java memory limit for jas3: The default -Xmx is set to 256M in jas3 script, this option needs to be modified for larger memory size. Rocks OS behavior: Rocks crashed if all the memory is used. 9

14 References [1] [2] An application of the CSSE code: analysis of pulse pile-up in NaI(Tl) detectors used for TNA, M.Palomba, G.D erasmo, A. Pantaleo, Nucl. Instr. and Meth. A 498 (2003) ; 10

15 Figure 3: Neutron fluence as function of the distance r defined in the previous report for the soil (left figure) and TNT (right figure). 11

16 Figure 4: Event-time distribution of the photons hitting the sensitive detectors. Thermal neutrons of less than 0.5 MeV have been cut-off for this figure. 12

17 x:y:z x y z Figure 5: Neutron distribution in the soil (plotted with Root). The cynlinder shape distribution is the reactions with the TNT. A version of this figure is available with more events; due to the size of the file it must be downloaded from projet with username Faust and password x1f5z

18 Figure 6: Neutron distribution in the soil: OpenGL views of figure 5 with different angles. The simulation used for this figure contains fewer events than in the preceding figure because it makes images easier to handle. 14

19 x x:y:fastflux {z> && z<=0} y stflux fa x x:y:fastflux {z>-50.5 && z<-50.0} y x fastflu x x:y:fastflux {z> && z<=-250.0} y x fastflu Figure 7: Neutron flux distribution in the soil near the 0 surface (top figure), near 5cm surface (middle figure) and near 30cm surface (bottom figure). The top part of this figure was produced with a limited number of events in order to reduce its size. The full version is available at projet with username Faust and password x1f5z

20 energy Counts 10 5 plot1 Entries Mean RMS Energy Figure 8: Analysis of energy distribution including pile-up effect on the detector. 16

21 UNCLASSIFIED SECURITY CLASSIFICATION OF FORM (highest classification of Title, Abstract, Keywords) DOCUMENT CONTROL DATA (Security classification of title, body of abstract and indexing annotation must be entered when the overall document is classified) 1. ORIGINATOR (the name and address of the organization preparing the document. Organizations for who the document was prepared, e.g. Establishment sponsoring a contractor's report, or tasking agency, are entered in Section 8.) ANS Technologies Inc. University of Montreal, Lab. Rene-J.A.-Levesque P.O. Box 6128, Station CV Montreal QC H3C 3J7 2. SECURITY CLASSIFICATION (overall security classification of the document, including special warning terms if applicable) UNCLASSIFIED 3. TITLE (the complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title). GEANT4 Particle Simulations in Support of the Neutron Interrogation Project 4. AUTHORS (Last name, first name, middle initial. If military, show rank, e.g. Doe, Maj. John E.) Atlantic Nuclear Services Ltd. 5. DATE OF PUBLICATION (month and year of publication of document) December a. NO. OF PAGES (total containing information, include Annexes, Appendices, etc) 16 6b. NO. OF REFS (total cited in document) 2 7. DESCRIPTIVE NOTES (the category of the document, e.g. technical report, technical note or memorandum. If appropriate, enter the type of report, e.g. interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.) Progress report 8. SPONSORING ACTIVITY (the name of the department project office or laboratory sponsoring the research and development. Include the address.) Defence R&D Canada Suffield, PO Box 4000, Station Main, Medicine Hat, Alberta, Canada T1A 8K6 9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.) 9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.) W R108 10a. ORIGINATOR'S DOCUMENT NUMBER (the official document number by which the document is identified by the originating activity. This number must be unique to this document.) DRDC Suffield CR b. OTHER DOCUMENT NOs. (Any other numbers which may be assigned this document either by the originator or by the sponsor.) 11. DOCUMENT AVAILABILITY (any limitations on further dissemination of the document, other than those imposed by security classification) ( x ) Unlimited distribution ( ) Distribution limited to defence departments and defence contractors; further distribution only as approved ( ) Distribution limited to defence departments and Canadian defence contractors; further distribution only as approved ( ) Distribution limited to government departments and agencies; further distribution only as approved ( ) Distribution limited to defence departments; further distribution only as approved ( ) Other (please specify): 12. DOCUMENT ANNOUNCEMENT (any limitation to the bibliographic announcement of this document. This will normally corresponded to the Document Availability (11). However, where further distribution (beyond the audience specified in 11) is possible, a wider announcement audience may be selected). UNCLASSIFIED SECURITY CLASSIFICATION OF FORM

22 UNCLASSIFIED SECURITY CLASSIFICATION OF FORM 13. ABSTRACT (a brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C) or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual). DRDC Suffield is developing nuclear-based technologies that will provide the Canadian Forces (CF) and other Public Security agencies with advanced explosives detection capabilities, such as to aid in the identification and evaluation of vehicle borne Improvised Explosive Devices (IEDs). As part of this effort, they require a high-precision, flexible neutron transport simulation program consistent with their existing analysis tools. Geant4 is a recently developed open-source C++ based radiation simulation package created by a world-wide collaboration of universities and provides a complete set of tools for all areas of detector simulation. As the Geant4 code base is relatively new and not widely used outside the physics community only a few attempts have been made to create a parallel version capable of working on a commodity component computer cluster. A previous contract, W R108, developed a parallel Geant4 simulation framework, TNAG4Sim, running on the DRDC Suffield parallel Linux cluster. However, that code suffered from an artificial limitation in total event number, limiting the statistical accuracy of the output data. This report provides an explanation and proposes a solution to the event limit problem. Data persistency was included using Reflex with ROOT I/O method, and data analysis tools written for extracting a ROOT readable files from the Reflex/ROOT format to an independent ROOT ntuple. Lastly, a pile-up analysis tool is provided. We report on a successful full simulation run, with a reasonable event number, for testing this new cluster configuration. 14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifies, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g. Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurusidentified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.) Simulation, GEANT4, Fast Neutron Analysis, Thermal Neutron Analysis, Explosive, Detection UNCLASSIFIED SECURITY CLASSIFICATION OF FORM

23

24 Defence R&D Canada Canada's Leader in Defence and National Security Science and Technology R & D pour la défense Canada Chef de file au Canada en matière de science et de technologie pour la défense et la sécurité nationale