Paralleliza(on Challenges for Ensemble Data Assimila(on

Transcription

1 Paralleliza(on Challenges for Ensemble Data Assimila(on Helen Kershaw Institute for Mathematics Applied to Geophysics, National Center for Atmospheric Research

2 What am I going to talk about? What s ensemble data assimilation? What s DART? What s parallel about DART? What s not so parallel about DART? Data decomposition IO Algorithm and communication Software engineering concerns

3 What s ensemble data assimila(on?

4 Ensemble Data Assimila(on group of model forecasts

5 Ensemble Data Assimila(on group of model forecasts Measurements

6 Ensemble Data Assimila(on group of model forecasts Measurements Improved estimate

7 What s DART?

8 DART is used at: Public domain sobware for Data Assimila(on Well- tested, portable, extensible, free! Models Toy to HUGE 43 UCAR member universi(es More than 100 other sites Observa(ons Real, synthe(c, novel An extensive Tutorial With examples, exercises, explana(ons People: The DAReS Team

9 The State

10 The State

11 The State

12 The State pressure temperature vapor mixing ratio

13 The State DART state vector pressure temperature vapor mixing ratio

14 The State pressure temperature vapor mixing ratio multiple copies

15 The State pressure temperature vapor mixing ratio multiple copies

16 The State pressure temperature vapor mixing ratio multiple copies

17 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

18 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

19 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

20 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

21 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

22 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

23 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

24 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

25 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

26 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

27 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

28 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

29 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

30 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

31 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

32 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

33 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

34 Assimilation Apply the updates in parallel Round robin layout for load balancing - localization

35 Data decomposi(ons Whole model state available to a single processor All copies of some variables available to a single processor

36 Data decomposi(ons Whole model state available to a single processor All copies of some variables available to a single processor

37 Data decomposi(ons Whole model state available to a single processor All copies of some variables available to a single processor

38 Why do we need to change anything?

39 What does DART look like in memory? Whole model state available to a single processor All copies of some variables available to a single processor

40 What does DART look like in memory? Ensemble size = 4 memory tasks have a whole copy of the model state Other tasks do not task

41 Why do we use this decomposi(on? Calculation of the forward operator

42 Why do we use this decomposi(on? Calculation of the forward operator What the model thinks the observation should be

43 Why do we use this decomposi(on? Calculation of the forward operator What the model thinks the observation should be

44 Why do we use this decomposi(on? Calculation of the forward operator What the model thinks the observation should be

45 Limita(ons of having these two decomposi(ons: Hard minimum on calculation time Hard maximum on model size You have to move all your data

46 Idea: Only use the assimilation decomposition

47 Idea: Only use the assimilation decomposition Use one sided communication to grab state elements when needed

48 Idea: Only use the assimilation decomposition Use one sided communication to grab state elements when needed Reduce data movement Removes hard memory limit

49 Idea: Only use the assimilation decomposition Use one sided communication to grab state elements when needed Reduce data movement Removes hard memory limit Vectorization of forward operator calculations

50 More scalable forward operator Memory

51 More scalable forward operator Memory

52 More scalable forward operator Memory

53 More scalable forward operator Memory Calculation 4 tasks doing all observations for 1 copy

54 More scalable forward operator Memory Calculation 4 tasks doing all observations for 1 copy Lots of tasks doing some observations for all copies

55 Lorenz_96 forward operator observations observations state complete distributed state 1500 state complete distributed state observations wall clock get obs ens (s) wall clock get obs ens (s) 10 1 core seconds get obs ens (s) number of processors number of processors number of processors wall clock core seconds

56 CAM FV forward operator Specific humidity only : observations processors state complete 1.01s 0.96s distributed state 0.73s 0.18s

57 WRF forward operator 54, 400 observations processors state complete 0.6s 0.6s distributed 2.0s 0.7s

58 IO

59 IO Models do not run ensemble complete 1 3 2

60 IO Models do not run ensemble complete 1 3 2

61 IO Models do not run ensemble complete 1 3 2

62 IO Models do not run ensemble complete 1 3 2

63 IO Models do not run ensemble complete You have to move data from the model to DART

64 IO Ideally:

65 IO Ideally: Never looks like this in memory

66 IO All DART requires is that there are multiple model forecasts

67 IO time Multiple model forecasts to create the ensemble

68 IO model run time Multiple model forecasts to create the ensemble

69 IO model run time Multiple model forecasts to create the ensemble

70 IO model run time Multiple model forecasts to create the ensemble

71 IO model run time Multiple model forecasts to create the ensemble

72 IO model run time when you can start DART Multiple model forecasts to create the ensemble

73 IO IO for each ensemble member time when you can start DART Multiple model forecasts to create the ensemble

74 IO ensemble members Model run ~1000 tasks time Multiple model forecasts to create the ensemble

75 IO ensemble members Model run ~1000 tasks when you can start DART time Multiple model forecasts to create the ensemble

76 IO ensemble members Model run ~1000 tasks when you can start DART time Multiple model forecasts to create the ensemble

77 IO ensemble members CESM Model run ~10000 tasks time Multi-instance forecasts to create the ensemble

78 IO ensemble members CESM Model run ~10000 tasks Restart files for each model time Multi-instance forecasts to create the ensemble

79 IO ensemble members CESM Model run ~10000 tasks Restart files for each model time Multi-instance forecasts to create the ensemble

80 IO Model run ~10000 tasks ensemble members CESM DART CESM DART CESM time Multi-instance forecasts to create the ensemble

81 IO Model run ~10000 tasks ensemble members CESM DART CESM DART CESM time Should the IO speed drive the data layout?

82 Algorithm choice and communica(on The forward operator parallelizes The assimilation parallelizes

83 Algorithm choice and communica(on The forward operator parallelizes The assimilation parallelizes Communication does not scale

84 Broadcasts do i = 1:number of observations 1 observation i = 1 end do

85 Broadcasts do i = 1:number of observations i = 1 1 owner end do

86 Broadcasts do i = 1:number of observations i = 1 1 end do

87 Broadcasts do i = 1:number of observations i = 2 2 end do

88 Broadcasts do i = 1:number of observations i = 3 3 end do

89 Broadcasts do i = 1:number of observations i = 4 4 end do

90 Broadcasts do i = 1:number of observations i = 5 5 end do

91 Broadcasts do i = 1:number of observations i = 6 6 end do

92 Broadcasts do i = 1:number of observations i = 7 7 end do

93 Broadcasts do i = 1:number of observations i = 8 8 end do

94 Further Complica(ons

95 Further Complica(ons Or, software engineering concerns

96 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is?

97 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is? Allow whole state to be stored if the memory is available

98 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is? Allow whole state to be stored if the memory is available Does this mean a vectorized and non-vectorized version of the forward operator for each model?

99 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is? Allow whole state to be stored if the memory is available Need to remain user extensible

100 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is? Allow whole state to be stored if the memory is available Need to remain user extensible Backward compatible?

101 Further Complica(ons Or, software engineering concerns What about all the users who are happy with DART as it is? Allow whole state to be stored if the memory is available Need to remain user extensible Backward compatible? Manageable code

102 Collaborators?

103 Learn more about DART at:

104 Parallel Observa(on Processing

105 Parallel Observa(on Processing

106 Parallel Observa(on Processing Uniform: 127,000 obs Radar: 25,000 obs. 0.2 Radar: 25,000 obs Satellite track: 25,000 obs.

107 Parallel Observa(on Processing Observations that are more than 0.05 apart are independent.

108 Parallel Observa(on Processing Find minimum number of subsets of independent observations Mutual exclusion scheduling problem Use greedy algorithm: Decreasing Greedy Mutual Exclusion (DGME)

109 Parallel Observa(on Processing Red shows observations in a given subset Observations in Color Observations in Color Irregular Observations -> Load Balance Challenges

110 Parallel Observa(on Processing Last subsets only have a few observations each. -These are in regions where satellite and radar overlapped. - May be significant load balance issue. 400 Observations per Color Color

111 Observations 1 December 2006 GPS ACARS and Aircraft Radiosondes Sat Winds

112 Parallel netcdf Can we use this to transpose during IO? Simple for DART restart files Not simple for model restart files

113 Parallel netcdf Can we use this to transpose during IO? Simple for DART restart files - stride through a vector Not simple for model restart files

114 Parallel netcdf Can we use this to transpose during IO? Simple for DART restart files - stride through a vector Not simple for model restart files - can t ignore the dimensionality of each variable

115 Parallel netcdf Can we use this to transpose during IO? Simple for DART restart files - stride through a vector Not simple for model restart files - can t ignore the dimensionality of each variable

116 Parallel netcdf Can we use this to transpose during IO? Simple for DART restart files - stride through a vector Not simple for model restart files - can t ignore the dimensionality of each variable Should the IO speed drive the assimilation data layout?

117 Irregular Observa(ons - > Load Balance Challenges Simulate performance for idealized observation set (2% of obs shown) Uniform: 127,000 obs Radar: 25,000 obs. 0.2 Radar: 25,000 obs Satellite track: 25,000 obs.

118 IO You need to run a bunch of model forecasts Convert the model output to DART format Do data assimilation with DART Convert back to model input

119 Calcula(on of the Forward Operator

120 Calcula(on of the Forward Operator grid points at each model level

121 Calcula(on of the Forward Operator Observation at a vertical location in pressure/height/ grid points at each model level

122 Calcula(on of the Forward Operator Observation at a vertical location in pressure/height/ grid points at each model level The variables in the state determine the location of the observation

123 Calcula(on of the Forward Operator Observation at a vertical location in pressure/height/ grid points at each model level The variables in the state determine the location of the observation Interpolate to find the expected value of the observation

124 But vectorization is not perfect: An observation can be in different model levels depending on the state Observation

125 What s parallel about DART?

126 First, look at the serial version of the algorithm observation and error variance ensemble approximation of the observation updates

127 Algorithm choice and communica(on

128 Algorithm choice and communica(on Broadcast

129 IO Worst-case scenario

130 IO You need to run a bunch of model forecasts write to file

131 IO You need to run a bunch of model forecasts write to file Convert the model output to DART format read from file write to file

132 IO You need to run a bunch of model forecasts write to file Convert the model output to DART format read from file write to file Do data assimilation with DART read from file write to file

133 IO You need to run a bunch of model forecasts write to file Convert the model output to DART format read from file write to file Do data assimilation with DART read from file write to file Convert back to model input read from file write to file

134 IO Models do not run ensemble complete You have to move data from the model to DART

135 IO Scripting Queuing Scaling

136 IO ensemble members CESM Model run ~10000 tasks Restart files for each model time Should the IO speed drive the data layout?

137 Nota(on

138 What s parallel about DART? observation and error variance ensemble approximation of the observation updates

139 Why do we need to change anything? Or, what s not so parallel about DART? Multiple data decompositions IO Algorithm choice and communication

140 Limita(ons of having these two decomposi(ons: The forward operator does not scale beyond processors = ensemble members Users have models that are too large to fit into the memory of a single node You have to transpose data between decompositions