CCSM Performance with the New Coupler, cpl6

Transcription

1 CCSM Performance with the New Coupler, cpl6 Tony Craig Brian Kauffman Tom Bettge National Center for Atmospheric Research Jay Larson Rob Jacob Everest Ong Argonne National Laboratory Chris Ding Helen He Lawrence Berkeley National Laboratory RIST Workshop, March 3-5, 2003, INGV, Rome, Italy

2 Topics CCSM overview cpl5 review cpl6 goals cpl6 design and datatypes cpl6 performance merging mapping communication Summary

3 CCSM Overview CCSM = Community Climate System Model (NCAR) Designed to evaluate and understand earth s global climate, both historical and future. Multiple executables (5) Atmosphere (CAM), MPI/OpenMP Ocean (POP), MPI Land (CLM2), MPI/OpenMP Sea Ice (CSIM4), MPI Coupler (CPL5), OpenMP

4 CCSM2 Hub and Spoke System ocn atm cpl ice lnd Each component is a separate executable Each component on a unique set of hardware processors All communications go through coupler Coupler communicates with all components maps (interpolates) data merges fields computes some fluxes has diagnostic, history, and restart capability

5 CCSM Platforms Currently support IBM Power3, Power4 SGI Origin O2k, O3K Nearly support HP/Compaq Future? Linux Vector Platform (Cray X1, NEC/Earth Simulator)

6 CCSM Resolution and Timing T42 resolution atm and land, 26 vertical levels in atm (128x64x26, 200k cells) 1 degree resolution ocean and ice, 40 vertical levels in ocean (320x384x40, 4M cells) On 100 processors of IBM power4 (8 processors/node, 1.3 Ghz clock, Colony): Model runs about 10 simulated years / day Requires about a month to run 300 years Science requirements set coupling frequency between models and data flow

7 CCSM Overview (part 2) F90 primarily Netcdf history files Binary restart files SCIDAC - DOE ESMF - NASA

8 cpl5 Shortcomings cpl5 is a shared memory application and uses Open_MP threading to achieve parallel computation efficiency cpl5 communication with models is not parallelized, does root-to-root communication only, requires gathers and scatters on distributed memory components Increasing resolutions or coupling frequencies could result in coupling performance bottlenecks cpl5 coupling is hard-wired to MPI and is not easily extensible as currently implemented

9 cpl6 Goals Create a fully parallel distributed memory coupler Implement M to N communication between components Improve communication performance to eliminate any potential future bottlenecks as a result of increased resolution Improve coupling interfaces, abstract communication method away from components Improve usability, flexibility, and extensibility of coupled system Improve overall performance

10 The Solution Build a new coupler framework with abstracted, parallel communication software in the foundation. Create a coupler application instantiation called cpl6 which reproduces the functionality of cpl5: cpl6 MCT* MPH** *Model Coupling Toolkit ** Multi-Component Handshaking Library

11 MCT: Model Coupling Toolkit www-unix.mcs.anl.gov/acpi/mct Major Attributes: Maintains model decomposition descriptors (e.g., global to local indexing) Inter- and intra- component communications and parallel data transfer (routing, rearranging) Flexible, extensible, indexible field storage Time averaging and accumulation Regridding (via sparse matrix vector multiply) MCT eases the construction of coupler computational cores and component-coupler interfaces.

12 MPH: Multi-Component Handshaking Library General Features: Built on MPI Establishes MPI communicator for each component Performs component name registration Allows resource allocation for each component Supports different execution modes MPH allows generalized communicator assignment, simplifying the component model communication and inter-component communication setup process.

13 cpl6 Architecture main program Layer 1a MCT wrapper control maindata msg map flux restart history diag Layer 1b coupling interface Layer 1c calendar utilities csmshare datatypes MCT derived objects MCT base objects MPEU utilities Layers 2-5 Vendor utilities

14 MCT Data Types Attribute Vector Fundamental data storage type 2d integer and real arrays (field,grid point) Strings for field names Global Seg Map Decomposition information Router M to N communication information Rearranger Local Communication information Smat Scattered mapping matrix data

15 cpl6 Data Types Infobuffer Vector of integers and reals, scalar data Domain cpl6 grid data type Name, Attribute Vector of grid data, GSMap Bundle Fundamental cpl6 storage data type, array data Name, Domain, Attribute Vector, Counter Contract Map Bundle, Infobuffer, Router Name, Smat, Domains, Rearranger

16 cpl6 Modules cpl_fields_mod Shared module, used by all components Sets field numbers and names Differentiates states and fluxes Naming convention allows automatic routing of data between components for simple fields cpl_interface_mod Simple interfaces, simple arguments Components only need to define contract data types. Within the interface, domains, routers, bundles, and contracts will be initialized on the component processors and used Components don t know about MCT, cpl6 data types, or the underlying communication method Extensible cpl6 design is more flexible and extensible than cpl5

17 cpl6 Design: Another view of CCSM hardware processors atm lnd ice ocn cpl coupling interface layer In cpl5, MPI was the coupling interface In cpl6, the coupler is now attached to each component Components unaware of coupling method Coupling work can be carried out on component processors Separate coupler no longer absolutely required

18 CCSM Performance: cpl5 vs cpl6 Merging Trivially parallel operation, cache usage important Production configuration timings Mapping Benchmarks tests for a2o, o2a mapping Effect of bundling multiple interpolations Comparison of mapping in production configuration Communication Focus on coupler to ice communication, high frequency, high resolution Unit tests and production configuration

19 Merging: cpl5 vs cpl6 Ocean Grid (122,880 points) 240 merging calls, 16 fields Production configuration secs cpl6 cpl number of pes

20 Merging Discussion Some serial performance optimization has been carried out in cpl6 cpl6 performs better than cpl5 for merging cpl6 scaling is not limited to a shared memory node cpl6 scaling is acceptable for trivially parallel operations Eliminated Open_MP overhead compared to cpl5 cpl6 will perform better than cpl5 and on a larger number of processors for simple parallel operations.

21 Mapping: cpl5 vs. cpl6 cpl5 mapping is a shared memory operation cpl6 mapping is currently distributed memory parallel only, and allows distributed, parallel mapping across several compute nodes cpl6 mapping requires MPI communication of data Rearrange data decomposition on source grid, then map to destination grid, OR. Map to destination grid, then rearrange to destination decomposition

22 secs Mapping: ocn -> atm Ocn (122,880 points) -> Atm (8192 points) 120 mapping calls, 9 fields number of pes cpl6 cpl5

23 Mapping: atm -> ocn effect of bundling mapping fields secs/field field 8 fields 16 fields field fields fields number of pes

24 secs Mapping: cpl5 vs cpl number of pes 10 simulated days Production configuration IBM-power4, 8 way nodes cpl6 cpl5

25 Mapping Discussion Mapping with cpl6 outperforms cpl5 at the same processor count in unit tests for a2o and o2a Bundling of fields for input into the mapping function is a clear winner over mapping single fields. Mapping performance is highly dependent upon (not shown): the grid sizes and shapes the data decomposition used the load imbalance created by the mapping procedure cpl6 mapping is slower or the same speed as cpl5 in comparisons on a production configuration. The cpl6 mapping method was originally designed to be easy to use, flexible, and extensible. As a result, there are special intermediate bundles and domains, extra array copies, associated rearranging, and array operations that do not use cache efficiently. We expect to improve mapping performance in the near future. cpl6 allows efficient, parallel, scalable mapping which can outperform cpl5 and provide flexibility for future CCSM configurations.

26 CCSM Communication: cpl5 vs cpl6 Coupler on 8pes Ice component on 16pes 240 transfers, 21 fields Production configuration cpl5 cpl6 copy=2.5s copy=1.3s copy=0.0s copy=0.0s comm=7.5s comm=5.0s comm=9.0s comm=8.0s gather =9.0s scatter =36.2s copy=1.0s copy=0.5s cpl5 communication=61.5s cpl6 communication=18.5s

27 cpl5 cpl6 Communication: cpl5 vs. cpl6 Coupler pes 1 (4) Ice pes 1 (16) cpl -> ice (secs) ice fields of size 122, transfers ice -> cpl (secs) apples and apples

28 Communication Discussion The cpl5 numbers illustrate the current CCSM2 performance on a single node using 4 pes and the ice model using 16 pes (4/16 configuration). Note that the cpl6<->ice single processor configuration simulates the root-to-root communication of cpl5<->ice in CCSM2.0. cpl6 is slower (1/1 configuration) due to the overhead created in pushing data thru the MCT. When cpl6 utilizes the full parallel capability of the 4/16 configuration (apples and apples), it clearly outperforms cpl5. cpl6 scales to multiple numbers of pes. cpl6 will be able to run on more pes than cpl5, will allow larger configurations of CCSM, and will improve communication performance.

29 Summary cpl6 is a distributed memory application, no threading implemented currently does M to N communication performance: Generally faster and better scaling than cpl5 Communication significantly faster than cpl5, potential important bottleneck eliminated Mapping in cpl6 (not MCT) requires further optimization is more flexible, usable and extensible scientific validation nearly complete, release expected soon