Evaluating the Performance of the Community Atmosphere Model at High Resolutions

Transcription

1 Evaluating the Performance of the Community Atmosphere Model at High Resolutions Soumi Manna MS candidate, University of Wyoming Mentor: Dr. Ben Jamroz National Center for Atmospheric Research Boulder, CO August 2, 2013

2 Overview Background Community Atmosphere Model (CAM5) High-Order Method Modeling Environment (HOMME) Space-filling curves (SFC) How well Space-filling curve work on refined and non refined mesh Performance statistics using Python Scalasca performance data Conclusion Future Work

3 CAM(Community Atmosphere Model) Developed primarily at NCAR for climate research communities One of multiple component models in the Community Earth System Model (CESM) 3

4 CAM(Community Atmosphere Model) Efforts focused on increasing resolution of CAM5 Use of mesh refinement in CAM5 through High-Order Method Modeling Environment (HOMME) dynamical core Allow for regions with extremely high-resolution Produce a challenge to the current parallel domain decomposition algorithm 4

5 Project Goals Analyze performance of HOMME on high and variable resolutions Investigate quality of domain decompositions produced by space-filling curve algorithms for refined and unrefined meshes Evaluate performance metrics of realistic simulations on these meshes using automatic trace analysis tool Scalasca 5

6 HOMME (High Order Method Modeling Environment) A scalable and efficient spectral-element-based atmospheric dynamical core ( Spectral Elements: A quadrilateral patch of gridpoints Elements are currently squares on a cube, projected onto a sphere using gnomonic projection 6

7 SFC (Space-Filling Curves) A curve whose range contains entire 2-dimensional unit square Hilbert Curve Peano Curve Hilbert-Peano Curve 7

8 Space-Filling Curve on Non Refined Mesh

9 SFC on Refined Mesh Existing algorithm for quasi-uniform was extended to refined mesh Performance has not been analyzed Elements get mapped to the closest point on the SFC Elements are in non uniform order and can be very close to each other Refined regions require a high resolution SFC Impact quality of the decomposition of the coarse region 9

10 Statistics for Measuring Quality of Domain Decomposition Load balancing Each Processor gets equal amount of work Communication pattern Maximum point to point communication Number of neighboring processes Total Edgecut Build communication matrix Calculate communication pattern Edgecut Neighbors 1 2 Maximum P2P Communication

11 Implementation Modify HOMME to output the space filling curve ordering for elements Write python program to analyze the quality of the domain decomposition Maximum Point to Point Communication Edge Cut Number of Neighbors Run profiling tool Scalasca to see the communication cost Correlate profile data with statistics 11

12 SFC on Quasi-uniform Maximum number of neighbors for Ne120 Fewer neighbors for even number of element per partition 12

13 SFC on Quasi-uniform Average number of neighbors for Ne120 Number of neighbors (latency) dominate communication 13

14 SFC on Quasi-uniform Ratio of maximum to average point to point communication for Ne120 Optimal Edgecut for even number of element per partition 14

15 Refined Mesh (ARM) Refined grid over Atmospheric Radiation Measurement [ARM] sites Incorporating ARM observations into climate simulation Alternative to nested models for regional climate 15

16 Refined Mesh (ARM) Refined grid over Atmospheric Radiation Measurement [ARM] sites Incorporating ARM observations into climate simulation Alternative to nested models for regional climate 16

17 SFC on Refined Mesh Average number of neighbors for Ne120 and ARM Number of neighbors does not vary much for ARM Larger average communication partners for Ne120 17

18 SFC on Refined Mesh Maximum number of neighbors for Ne120 and ARM Maximum neighbors is higher for refined mesh 18

19 SFC on Refined Mesh Larger P2P communication for refined mesh Increases bandwidth cost 19

20 SFC on Refined Mesh gives Discontinuous Decomposition Increases number of neighbors (latency cost) Increases P2P communication (bandwidth cost) 20

21 SFC on Refined Mesh gives Discontinuous Decomposition Increases number of neighbors (latency cost) Increases P2P communication (bandwidth cost) 21

22 Scalasca Performance Data Used profiling tool Scalasca to measure communication Realistic atmosphere simulation on 300 MPI processes Calculated statistics accurately predicts amount of communication 22

23 Communication time does not agree with estimation Calculated communication statistics accurately predict communication pattern Expect communication time of simulations to agree with our statistics Unfortunately we do not see this 23

24 Communication Time Although we accurately predict communication pattern, communication time is erratic No spatial pattern independent of refined mesh 24

25 Node Dependent Communication Time Some nodes have wait time 5x Without this imbalance simulation runs 40% faster 25

26 Conclusion Quasi-uniform mesh: Fewer neighbors for even number of element per partition Number of neighbors (latency) dominate communication Optimal Edgecut for factor of two Refined mesh: Number of neighbors does not vary much SFC gives Discontinuous decomposition Increases the max number of neighbors(latency) Increases the interprocess edgecut (bandwidth) Statistics accurately calculates communication pattern Latency between nodes dominates communication cost 26

27 Future Work Use different machine to analyze performance data Look at the performance of modification of existing space-filling curve Investigate different partitioning methods 27

28 References

29 1. Dr. Ben Jamroz 2. Dr. John Dennis 3. All ASAP group members 4. Jennifer Williamson 5. Kristin Mooney 6. SIParCS 2013 Interns 7. NCAR/UCAR 8. University of Colorado 9. University of Wyoming Acknowledgement 29

30 Thank You Contact: Soumi Manna MS Student University of Wyoming 30