Fourteen years of Cactus Community Frank Löffler Center for Computation and Technology Louisiana State University, Baton Rouge, LA September 6th 2012
Outline Motivation scenario from Astrophysics Cactus structure technically Cactus structure socially Future directions Frank Lo ffler Fourteen years of Cactus Community 2012-09-06
Challenging Astrophysics Problems Black Holes and Neutron Stars Supernovae Cosmology Gravitational Wave Data Analysis Frank Lo ffler Fourteen years of Cactus Community 2012-09-06
Gravitational Wave Physics
Solving Einstein s Equations Einstein equations: G µν = 8πT µν 12 fully 2nd order PDE evolution equations 4 coupled constraint equations 4 gauge conditions GR hydrodynamics, MHD, radiation transport Fully numerical 3D models needed
Computational Requirements Unigrid scales to hundreds of thousands of cores Productions runs use 10 levels of mesh refinement, nested grids of size 60x60x60 Current mesh refinement runs scale up to 10k cores Runtime from weeks to few months
Challenge Many scientific/engineering components Physics Mathematics CFD Many numerical algorithm components Finite difference, finite volume, spectral methods Structured or unstructured meshes, mesh refinements Multipatch and multimodel Many different computational components Parallelism (MPI, OpenMP,...) Parallel I/O (e.g. Checkpointing) Visualization Challenge Defining good abstractions to bring these together in a unified, scalable framework, enabling science
Cactus Framework Structure
Cactus Core: The Flesh ANSI-C and Perl, C++ Independent of all other components Unified error handling Build system Parameter parsing/steering Global variable management Rule-based scheduler Extensible APIs Parallel Operations Input/Output Reduction Interpolation Timers Functionality provided by (swappable) components
Cactus Components: Thorns C, C++, Fortran 77, Fortran 90 Typically not implementing grid setup and memory allocation input/output interpolation, reductions Encapsulating some functionality initial data boundary conditions evolution systems equations of state remote steering (e.g. https server)
Basis Module Overview Basis for scalable algorithm development Most used: finite differences on structured meshes Parallel driver components Simple Unigrid Carpet: Multipatch, Mesh-refinement Method of lines Interfaces to external Libraries/Tools Interface to elliptic solvers (e.g. PETSc, Lorene) Input/Output: HDF5 Visualization: VisIt, OpenDX, Vish Other: PAPI, Hypre, Saga, Flickr, Twitter
Convenience Tools GetComponents Simfactory Formaline
Cactus as growing project Cactus
Cactus as growing project Cactus
Cactus as growing project Cactus
Cactus as growing project Cactus
Cactus as growing project Cactus
Social structure Few Core Members: Gabrielle Allen, Steven R. Brandt, Frank Löffler, Erik Schnetter,... Developers: about 50 worldwide Many more users Cactus Community Open Source Yearly releases Mailing lists Issue tracker IRC support channel Tutorials Web, HPC allocations, Repositories,...
Guiding Principles Open, community-driven software development Separation of physics software and computational infrastructure Stable interfaces, allowing extensions Simplify usage where possible: Doing science >> Running a simulation Students need to know a lot about physics (meaningful initial conditions, numerical stability, accuracy/resolution, have patience, have curiosity, develop a gut feeling for what is right...) Cactus Toolkit cannot give that, however: Open codes that are easy to use allow to concentrate on these things!
Credits, Citations In academics: citations, citations, citations! In Cactus: Open and free source No requirement to cite anything However: requested to cite a few publications Which publications: Few for the Cactus framework Some components list a few as well List published on website and manage through publication database
Future From certain to more speculative: Multiblock techniques GPU support Requirement-based scheduling Discontinuous Galerkin instead of finite differences Support for unstructured grids Completely requirement-based programming (MPI ParalleX?)
Tools: GetComponents Task: Collect software from various repositories at different sites Example simulation assembly: Cactus Flesh and Toolkit (svn.cactuscode.org) Core Einstein Toolkit (svn.einsteintoolkit.org) Carpet AMR (carpetcode.org, hg) Tools, Parameter Files and Data (svn.einsteintoolkit.org) Group Modules (x.groupthorns.org) Individual Modules (x.mythorns.org) x: cvs, svn, darcs, git, hg, http
Tools: Simulation Factory http://www.simfactory.org/ Task: Provide support for common, repetitive steps: Access remote systems, synchronize source code trees Configure and build on different systems semi-automatically Provide maintained list of supercomputer configurations Manage simulations (follow best practices, avoid human errors)
Tools: Formaline Task: Ensure that simulations are and remain repeatable, remember exactly how they were performed Take snapshots of source code, system configuration; store it in executable and/or git repository Tag all output files