Introduction to the Einstein Toolkit Details

Size: px

Start display at page:

Download "Introduction to the Einstein Toolkit Details"

Edward Hardy
5 years ago
Views:

1 Introduction to the Einstein Toolkit Details Roland Haas TAPIR, Caltech July 26 th 2013

2 Einstein Toolkit main components Toolkit is based on three core components Cactus GetComponents provides the numerical framework and defines interfaces for codes from different authors to interact with each other downloads modules the make up a simulation code Simulation Factory unified user interface to start and manage simulations on a variety of machines from laptops to supercomputers

3 What is Cactus? Cactus is a framework for developing portable, modular applications

4 What is Cactus? Cactus is a framework for developing portable, modular applications focusing, although not eclusively, on high-performance simulation codes he other modules

5 What is Cactus? Cactus is a framework for developing portable, modular applications focusing, although not eclusively, on high-performance simulation codes he other modules separate concerns: designed to allow eperts in different fields to develop modules based upon their eperience and to use modules developed by eperts in other fields with minimal knowledge of the internals or operation of the other modules

6 What is Cactus? Cactus is a framework for developing portable, modular applications focusing, although not eclusively, on high-performance simulation codes he other modules separate concerns: designed to allow eperts in different fields to develop modules based upon their eperience and to use modules developed by eperts in other fields with minimal knowledge of the internals or operation of the other modules mature 12 years of development

7 Cactus Goals Portable Different development machines Different production machines

8 Cactus Goals Portable Different development machines Different production machines Modular Standard interfaces for module interaction for easier code interaction, writing and debugging Interchangeable modules with same functionality

9 Cactus Goals Portable Different development machines Different production machines Modular Standard interfaces for module interaction for easier code interaction, writing and debugging Interchangeable modules with same functionality Easy to use Good documentation Try to let users program the way they are used to Support all major (HPC) programming languages

10 Flesh The flesh is the central component of Cactus. It interfaces with modular components called thorns. The flesh provides: Variables & Data Types Parameters Functions for: Parallelisation Input/Output Coordinates Reduction Interpolation Information Staggering Indeing Ghostzones

11 Thorns Some thorns provide additional functionality, while others serve as applications. Thorns are grouped into arrangements which supply some common functionality. Eample thorns: EinsteinInitialData TwoPunctures TOVSolver Carpet Carpet CarpetIOHDF5 EinsteinAnalysis WeylScal4 Multipole initial data initial data for binary black holes initial data for single spherical star adaptive mesh refinement driver simulation driver, adaptive mesh refinement 1d, 2d, 3d output to HDF5 etract data from simulation computes gravitational wave content decompose into spherical harmonic modes

12 Directory structure Cactus -- doc -- UsersGuide.pdf -- ReferenceManual.pdf -- configs -- sim -- sim-debug -- src -- repos -- simfactory -- arrangements -- EinsteinInitialData -- TwoPunctures -- Carpet -- CarpetIOHDF5

13 GetComponents why? 100 thorns used for simulation Thorns come from a variety of sources: core Cactus thorns, ET thorns, AEIthorns, LSUThorn, Carpet, github.com, bitbucket.org Thorns come in a variety of formats: git, svn, cvs, darcs,... Multiple access levels: publicly available, password required Multiple thorns per repository? Separate documentation? Eample files? Specific code version, release date

14 GetComponents what? (parallised) download of modules from source code repositories access to sub-parts of repositories assemble directory structure simplify update of modules show local changes in all thorns operates on ThornList describing collection of thorns

15 GetComponents how?!crl VERSION = 1.0!DEFINE ROOT = Cactus!DEFINE ARR = $ROOT/arrangements!TARGET = $ROOT!TYPE = svn!auth URL = = Cactus!NAME =.!TARGET = $ARR!TYPE = git!auth_url = carpetgit@carpetcode.org:carpet!url = git://carpetcode.org/carpet!checkout = Carpet/doc Carpet/Carpet Carpet/CarpetIOHDF5

Captures the best practices of eperienced users, ensuring repeatable and well-documented scientific results.

16 Simulation Factory The Simulation Factory: Command-line tools for setting up Cactus distribution and managing simulations on a variety of supercomputers, including most of the XSEDE machines. Captures the best practices of eperienced users, ensuring repeatable and well-documented scientific results. Developed by Erik Schnetter and Michael Thomas. Included in virtual machine: Cactus/simfactory. For more, see: More on Simfactory in the hands-on session.

17 Cactus thorn writing: Cactus concepts thorn Cactus module providing physics or infrastructure capabilities grid function discrete representation of function eg. density ρ(, y, z) on the numerical grid, eg. rho i,j,k grid scalar single number (or set of numbers) managed by Cactus, eg. total mass schedule order in which Cactus calls user supplied routines, defined through orderign relations SCHEDULE compute rho BEFORE compute mass parameter user specifyable option controlling simulation. Used for eample to provide central density in TOV stars, give final time of simulation, specify type of equation of state parameter file set of parameter values used in a single simulation run, key value pairs

18 Cactus thorn writing: anatomy of a thorn Directory structure: GRhydro/ -- COPYRIGHT -- README -- configuration.ccl -- doc -- documentation.te -- interface.ccl -- schedule.ccl -- param.ccl -- src -- GRHydro Startup.F90, make.code.defn

19 configuration.ccl configuration.ccl # Configuration definition for thorn GRHydro REQUIRES EOS_Omni Boundary CartGrid3D SpaceMask ADMMacros OPTIONAL_IFACTIVE Carpet { } List the thorns without which yours will not compile. Popular choices include eternal libraries (GSL/LAPACK/LORENE/HDF5, etc.), EOS Omni for matter.

20 interface.ccl interface.ccl implements: GRHydro inherits: ADMBase, SpaceMask, Tmunubase, HydroBase USES INCLUDE HEADER: carpet.hh void FUNCTION SpatialDet(CCTK_REAL IN g, CCTK_REAL IN gy, \ CCTK_REAL IN gz, CCTK_REAL IN gyy, \ CCTK_REAL IN gyz, CCTK_REAL IN gzz, \ CCTK_REAL OUT det) PROVIDES FUNCTION SpatialDet WITH SpatialDeterminant LANG Fortran public: real dens type = GF Timelevels = 3 \ tensortypealias="scalar" interpolator="matter" \ "generalized particle number"

21 param.ccl param.ccl REAL rho abs min "A minimum rho below which evolution is \ turned off (atmosphere)." STEERABLE=recover { -1.0 :: "use relative minimum" 0.0:* :: "reset rho when it falls below" } -1.0 SHARES: HydroBase USES CCTK INT timelevels EXTENDS KEYWORD evolution method { "GRHydro" :: "Use GRHydro to evolve the hydro variables" } Parameters are typically set at startup, not during eecution.

22 schedule.ccl schedule.ccl storage STORAGE: dens[timelevels] if(cctk Equals(Bvec evolution method,"grhydro")) { STORAGE: HydroBase::Bvec[timelevels] STORAGE: GRHydro::Bcons[timelevels] STORAGE: Bconsrhs } schedule.ccl handles global storage control.

23 schedule.ccl schedule.ccl scheduling schedule Conservative2Primitive IN HydroBase Con2Prim { LANG: Fortran } "Convert back to primitive variables (general)" schedule GRHydro Boundaries IN HydroBase Select Boundaries \ AS GRHydro Bound { LANG: Fortran OPTIONS: LEVEL SYNC: dens SYNC: HydroBase::Bvec } "Select GRHydro boundary conditions" schedule.ccl handles subroutine scheduling.

24 schedule.ccl cont d schedule.ccl scheduling schedule group GRHydroRHS IN HydroBase RHS { STORAGE: GRHydro scalars } "Calculate the update terms" schedule SourceTerms IN GRHydroRHS BEFORE FluTerms { LANG: Fortran } "Source term calculation" schedule UpdateAtmosphereMask IN GRHydroRHS AFTER FluTerms { LANG: Fortran } "Alter the update terms if inside the atmosphere region"

25 make.code.defn files to compile # Source files in this directory SRCS = GRHydro_Source.F90 \ GRHydro_UpdateMask.F90 \ GRHydro_Prim2Con.c \ GRHydro_Boundaries.F90 SUBDIRS = make.code.defn

26 Final tips Plagiarize and copy liberally, especially at first!!! Consider compilation time when you write your thorn. You want a thornlist that is as small as possible, but not smaller. Get version control running first! Learn how to output. It is a huge timesaver for debugging/running.

27 Worked out eample: solving scalar wave equation For a given source function S(, y, t) find a scalar wave field ϕ(, y, t) inside the domain D with a boundary condition: inside D: D D 2 ϕ t 2 = c2 ϕ + S on the boundary D: ϕ D = 0

28 WaveToy Thorn: Discretization Discretization: approimating continuous function ϕ(, y, t) with a grid function ϕ (n) i,j : ϕ (n) i 1 ϕ (n+1) i ϕ (n) i ϕ (n 1) i ϕ (n) i+1 ϕ (n+1) i,j 2ϕ (n) i,j 2 t 2 2 ϕ t 2 = c2 ( 2 ϕ + 2 y ϕ) + S (c 1) + ϕ (n 1) i,j = ϕ (n) i+1,j 2ϕ(n) i,j + ϕ (n) i 1,j ϕ (n) i,j+1 2ϕ(n) i,j + ϕ (n) i,j 1 2 y 2 + S (n) i,j

29 WaveToyMoL Thorn Thorn structure: interface.ccl grid function phi[3]: phi phi p phi p p ϕ (n) ϕ (n 1) ϕ (n 2) Boundary SelectVarForBC param.ccl Parameters of initial Gaussian pulse: amplitude A, radius R, width σ schedule.ccl WaveToy InitialData WaveToy Evolution WaveToy Boundaries

30 WaveToy Thorn: Algorithm Illustration Grid structure GF allocation t Set up coords Compute t Initial data Rotate timelevels Evolve GF, sync Apply BCs Output data

31 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data Rotate timelevels Evolve GF, sync Apply BCs Output data

32 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data, phi, phi p, phi p p, phi, phi p, phi p p Rotate timelevels Evolve GF, sync Apply BCs Output data

33 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi, phi p, phi p p phi, phi p, phi p p Rotate timelevels Evolve GF, sync Apply BCs Output data

34 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data t = λ min phi, phi p, phi p p phi, phi p, phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

35 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

36 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

37 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

38 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

39 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

40 WaveToy Thorn: Algorithm Illustration t PROC0 PROC1 Grid structure GF allocation Set up coords Compute t Initial data phi phi p phi p p t = 0 Rotate timelevels Evolve GF, sync Apply BCs Output data

41 WaveToy Thorn: Algorithm Illustration t phi phi p phi p p PROC0 PROC t = 0 Grid structure GF allocation Set up coords Compute t Initial data Rotate timelevels Evolve GF, sync Apply BCs Output data

42 WaveToy Thorn Directory structure: WaveToy/ -- COPYRIGHT -- README -- configuration.ccl -- doc -- documentation.te -- interface.ccl -- schedule.ccl -- param.ccl -- src -- WaveToy.c -- make.code.defn

43 WaveToy Thorn Directory structure: WaveToy/ -- COPYRIGHT -- README -- configuration.ccl -- doc -- documentation.te -- interface.ccl -- schedule.ccl -- param.ccl -- src -- WaveToy.c -- make.code.defn

44 WaveToy Thorn interface.ccl: IMPLEMENTS: wavetoy_simple INHERITS: grid PUBLIC: CCTK_REAL scalarevolve TYPE=gf TIMELEVELS=3 { phi } "The evolved scalar field" CCTK_INT FUNCTION Boundary_SelectVarForBC( \ CCTK_POINTER_TO_CONST IN GH, CCTK_INT IN faces, \ CCTK_INT IN boundary_width, CCTK_INT IN table_handle, \ CCTK_STRING IN var_name, CCTK_STRING IN bc_name) REQUIRES FUNCTION Boundary_SelectVarForBC

45 WaveToy Thorn cont. schedule.ccl: STORAGE: scalarevolve[3] SCHEDULE WaveToy_InitialData AT CCTK_INITIAL { LANG: C } "Initial data for 3D wave equation" SCHEDULE WaveToy_Evolution AT CCTK_EVOL { LANG: C SYNC: scalarevolve } "Evolution of 3D wave equation" SCHEDULE WaveToy_Boundaries AT CCTK_EVOL AFTER WaveToy_Evolution { LANG: C } "Select boundary conditions for the evolved scalar" SCHEDULE GROUP ApplyBCs as WaveToy_ApplyBCs AT CCTK_EVOL AFTER WaveToy_Boundaries { } "Apply boundary conditions"

46 WaveToy Thorn cont. param.ccl: CCTK_REAL amplitude "The amplitude of the waves" { *:* :: "Anything" } 1.0 CCTK_REAL radius "The radius of the gaussian wave" { 0:* :: "Positive" } 0.0 CCTK_REAL sigma "The sigma for the gaussian wave" { 0:* :: "Positive" } 0.1

47 WaveToy Thorn cont. Directory structure: WaveToy/ -- COPYRIGHT -- README -- configuration.ccl -- doc -- documentation.te -- interface.ccl -- schedule.ccl -- param.ccl -- src -- WaveToy.c -- make.code.defn

48 WaveToy Thorn cont. WaveToy.c: void WaveToy Evolution(CCTK ARGUMENTS) { DECLARE CCTK ARGUMENTS; CCTK REAL dt2,d2i,dy2i,dz2i; dt2 = CCTK DELTA TIME*CCTK DELTA SPACE(2); d2i = 1.0/(CCTK DELTA SPACE(0)*CCTK DELTA SPACE(0)); dy2i = 1.0/(CCTK DELTA SPACE(1)*CCTK DELTA SPACE(1)); dz2i = 1.0/(CCTK DELTA SPACE(2)*CCTK DELTA SPACE(2)); /* Do the evolution */ for (int k=1; k<cctk lsh[2]-1; k++) { for (int j=1; j<cctk lsh[1]-1; j++) { for (int i=1; i<cctk lsh[0]-1; i++) { phi[cctk GFINDEX3D(cctkGH,i,j,k)] = 2*(1 - (dt2)*(d2i + dy2i + dz2i)) * phi p[cctk GFINDEX3D(cctkGH,i,j,k)] - phi p p[cctk GFINDEX3D(cctkGH,i,j,k)] + (dt2) * ( ( phi p[cctk GFINDEX3D(cctkGH,i+1,j,k )] +phi p[cctk GFINDEX3D(cctkGH,i-1,j,k )] )*d2i +( phi p[cctk GFINDEX3D(cctkGH,i,j+1,k )] +phi p[cctk GFINDEX3D(cctkGH,i,j-1,k )] )*dy2i +( phi p[cctk GFINDEX3D(cctkGH,i,j,k+1)] +phi p[cctk GFINDEX3D(cctkGH,i,j, k-1)] )*dz2i); } } } }

49 WaveToy Thorn cont. WaveToy.c cont.: void WaveToy InitialData(CCTK ARGUMENTS) { DECLARE CCTK ARGUMENTS; DECLARE CCTK PARAMETERS; for(int k=0; k<cctk lsh[2]; k++) { for(int j=0; j<cctk lsh[1]; j++) { for(int i=0; i<cctk lsh[0]; i++) { int vinde = CCTK GFINDEX3D(cctkGH,i,j,k); CCTK REAL dt = CCTK DELTA TIME; CCTK REAL X = [vinde]; CCTK REAL Y = y[vinde]; CCTK REAL Z = z[vinde]; CCTK REAL R = sqrt(x*x + Y*Y + Z*Z); phi[vinde] = amplitude*ep( - SQ( (R - radius) / sigma ) ); if (R == 0.0) { phi p[vinde] = amplitude*( *dt*dt/sigma)*ep(-dt*dt/sigma); } else { phi p[vinde] = amplitude/2.0*(r-dt)/r* ep( - SQ( (R - radius - dt)/ sigma ) ) + amplitude/2.0*(r+dt)/r* ep( - SQ( (R - radius + dt)/ sigma ) ); } } } } }

50 WaveToy Thorn cont. WaveToy.c: void WaveToy Boundaries(CCTK ARGUMENTS) { DECLARE CCTK ARGUMENTS; /* Uses all default arguments, so invalid table handle -1 can be passed */ if (Boundary SelectVarForBC (cctkgh, CCTK ALL FACES, 1, -1, "wavetoy simple::phi", "scalar") < 0) { CCTK WARN (0, "WaveToy Boundaries: Error selecting boundary condition"); } } make.config.defn: SRCS = WaveToy.c

51 Parameter file Eample parameter file: Cactus::cctk_run_title = "Simple WaveToy" ActiveThorns = "Time Boundary Carpet CarpetLib CartGrid3D CoordBase IOUtil CarpetIOBasic CarpetIOASCII CarpetIOHDF5 SymBase WaveToy" Cactus::cctk_itlast = Time::dtfac = 0.5 IO::out_dir = $parfile IOBasic::outInfo_every = 1 IOASCII::out1D_every = 10 IOASCII::out1D_vars = "wavetoy_simple::phi" IOHdf5::out_every = 100 IOHdf5::out_vars = "wavetoy_simple::phi"

52 Eercise: compute wave energy compute energy density ρ E = φ 2 d dy dz compute integral as sum reduction of IOScalar::out reductions during output

Introduction to the Cactus Framework

Introduction to the Cactus Framework The Cactus team Oct 25 2011 1 Introduction Outline 2 The Cactus Computational Toolkit Overview Capabilities 3 Visualization 4 Cactus Structure Overview The Flesh The