Direct Numerical Simulation of Turbulent Boundary Layers at High Reynolds Numbers.

Transcription

1 Direct Numerical Simulation of Turbulent Boundary Layers at High Reynolds Numbers. G. Borrell, J.A. Sillero and J. Jiménez, Corresponding author: School of Aeronautics, Universidad Politécnica de Madrid, 284 Madrid, Spain. Center for Turbulence Research, Stanford University, Stanford, CA 9435, USA. Abstract: A new high-resolution code for the direct numerical simulation of zero pressure gradient turbulent boundary layers over a flat plate has been developed. Its purpose is to simulate a wide range of Reynolds numbers from Re θ =3 to 68 while showing a linear weak scaling up to cores in the BG/P architecture. Special attention has been payed to the generation of proper inflow boundary conditions. The results are in good agreement with existing numerical and experimental data sets. Keywords: DNS, Boundary Layer, Turbulence. Introduction. Turbulent boundary layers have an undeniable technological interest. Roughly half of the energy dissipation due to the movement of vehicles and transport through pipes is caused by or due to the presence of a turbulent boundary layer. This is the reason why turbulent boundary layers were among the first flows to be simulated[]. Our current research is focused on understanding the flow in the turbulent regions that are far from the wall where the Reynolds number plays a significant role. Our approach is to analyze data obtained from DNS simulations. While high-quality simulations exist for other wall bounded flows, mainly channels; similar resources are not available for boundary layers yet. We decided to generate them. We needed a high resolution code that is able to perform a DNS simulation of a boundary layer that has good performance and excellent scalability. The starting point is the code described in [4] but the result has seen severe modifications. Two simulations using this code are running at this moment in two BG/P supercomputers. A zero pressure gradient boundary layer over a flat plate with Re θ = 68 is running in Intrepid at Argonne National Laboratory and a forced boundary layer with artificial roughness with Re θ = 3 42 is running in Jugene at Jülich Forschungszentrum. The aggregated computational cost of these two simulations is roughly Million CPU hours.. Computational setup. Turbulent Boundary Layers are harder to compute than other wall parallel flows like channels because they have only one homogeneous direction and they are not universal. The non universality comes from

2 the fact that the boundary conditions in the inflow and wall-normal directions have to model a semi infinite domain. A schema of the computational domain can be seen in figure. Figure : Scheme of the computational domain and boundary conditions. The x, y, and z axis correspond to the streamwise, wall-normal and spanwise directions respectively. The integration method is a primitive-variable finite difference fractional-step. The temporal integration is a low-storage third order Runge-Kutta in which wall-normal second derivative terms use a Crank-Nicholson scheme to increase the time step. The spatial discretization is a fourth-order compact finite differences in the x and y directions and spectral Fourier in the z direction. The simulation is split in two concatenated domains with different boundary conditions. The planes π i and π i are given inflow boundary conditions, and outflow boundary conditions are assigned to π e and π e. The boundary conditions in π t and π t impose a zero pressure gradient on the domain. Finally, the spanwise direction is considered periodic. The mission of the first boundary layer (BL) is to provide accurate inflow boundary conditions to the second one (BL2). The inflow of BL is obtained from its own plane π that is rescaled using a method based on the one proposed by Lund, Wu and Squires[2]. The physical length of BL is chosen long enough to let the large scales recover from an unrealistic initial condition and, once this asymptotic state has been reached, the plane π 2 is used to give BL2 its inflow boundary condition. In consequence, a small portion of the BL simulation is thrown away. Given that the goal of BL is to allow the large scales to reach their asymptotic state and, given that the smaller scales take much shorter to reach a similar condition, BL is run at a coarser resolution than BL2. This setup allows to compute a single boundary layer with significantly less computational work..2 DNS and high order differentiation schemes. The goal of the present simulation is to achieve a Reynolds number based on the friction velocity up to Re τ = 2, a measure of the difference in size between the smallest and the largest scales present in wall bounded turbulent flows. To achieve a simulation up to Re τ = 2 a computational box of size is required with a total of 45 9 points per variable. In consequence it is crucial to optimize the memory usage. Being this a DNS simulation, the separation between adjacent collocation points is determined by the accuracy of the spatial discretization scheme and the local Kolmogorov scale. This scale changes depending on the distance to the wall, hence using a non uniform mesh in the wall normal direction is essential to save memory. Another important design decision is the order of the spatial discretization. Taking into account the modified wave number analysis, fourth order compact need the half of points than, for example, second order finite differences in each spatial coordinate. While a second order discretization scheme would probably get a higher flops count, it needs about 8 times more memory and twice the time steps.

3 2 Domain decomposition and communications. To take advantage of the distributed memory architectures, the computational domain must be partitioned. The only possible decomposition that guaranteed portability to the Blue Gene/P architecture was to use cross-stream planes schematized in the figure 2 as π i Figure 2: Elemental domains of the domain decomposition. To compute interpolations and derivatives over the x coordinate it is necessary to transpose the whole variable. This operation creates another elemental domain partition formed by lines in the streamwise direction, labeled in the figure 2 as ϖ i. Once these computations are finished the result is transposed back to planes π i. This global transpose is the bottleneck of the whole computational algorithm and has been carefully tuned. 2. The Blue Gene/P architecture. The most severe restriction of the Blue Gene/P architecture is the amount of memory installed per node being only 2 GiB. Taking the largest case to be run, only two π i planes fit in a single node. Given that every node has 4 processors, OpenMP is used to take advantage of all the resources of the node efficiently. Another important benefit from this approach is that the number of MPI processes is four times smaller than a MPI-only paradigm reducing the risk of performance degradation due to latency. The OpenMP runtime implementation available in the Blue Gene/P has, in our case, an efficiency of.92 running with all the available cores. It is important to emphasize that the use of a hybrid parallelism approach in the implementation is not a performance driven decision, but a mandatory feature to overcome the limitations of the architecture. 3 Performance and scalability. All the simulations run show a linear weak scaling up to cores and the same code without modifications is expected to scale further. The figure 3a shows that the communications time is typically 44% of the total run time, and that both computation and communications are scaling as expected. It is important to mention from figure 3b that the linear scaling is kept even when the MPI message size is close to the kb limit. This is an improvement from previous experiences with other network architectures where the latency is noticed below 3kB. Another interesting feature introduced in the code is parallel I/O operations using the parallel HDF5 library, now present in almost every supercomputer. Read and write operations on the Blue Gene s gpfs scale properly with the number of I/O nodes. The collective reading has achieved aggregated speeds of 25 GB/s, but collective writing is consistently two orders of magnitude slower.

4 -2 a) 2 b) Total Com m unications Time permessage [s] -3 Time [s] Size of the message [Bytes] Millions of points pernode Figure 3: (a) Scalability of total time and communication time for different test cases. The solid lines are linear regressions computed before taking logarithms of both axis. (b) Latency analysis. The solid line is a linear regression computed before taking logarithms of both axis. 4 Conclusions and future work. The code is giving results comparable to previous DNS simulations and experiments at similar Reynolds numbers. Figures 4, a-d are the mean and fluctuations of the velocity profiles of the present simulation compared with other available experimental [5],[6] and numerical [7] data sets. The contours of the last figure compare the obtained results with a channel simulation [3]. Some more work on tuning has been explored but a small speedup is expected. For example, overlapping computation and communications won t increase the performance more than 5%. This research is funded by the DoE s Incite program, the Prace initiative, CICYT TRA and Consolider CSD27-5. References [] P. R. Spalart. (988). Direct simulation of a turbulent boundary layer up to Re θ = 4. J. Fluid Mech. 87: 6-98 [2] T. S. Lund, X. Wu and K. D. Squires. (998) Generation or turbulent inflow data for spatiallydeveloping boundary layer simulations. J. Comput. Phys., 4: [3] S. Hoyas and J. Jiménez. (26) Scaling of the velocity fluctuations in turbulent channels up to Re τ = 23. Phys.Fluids, 8:72. [4] M. P. Simens, J. Jiménez, S. Hoyas and Y. Mizuno. (29) A high-resolution code for turbulent boundary layers. J. Comput. Phys., 228: [5] D. B. De Graaf and J. K. Eaton (2) Reynolds number scaling of the flat-plate turbulent boundary layer J. Fluid Mech. 422: [6] J. M Österlund, A. V Johansson, H. M. Nagib and M. Hites (2) A note on the overlap region in turbulent boundary layers Phys. Fluids 2: -4. [7] P. Schlatter and R. Örlü. (2) Assessment of direct numerical simulation data of turbulent boundary layers. J. Fluid Mech. 659: 6-26

5 U + v + (a) y + 3 (c).5.5 w u (b) 2 y + 3 (d) 2 3 y y + 3 λ z λ x + Figure 4: Experiments by [5],, Re θ = 526; and [6],, Re θ = 556. Simulations by [7], - -, Re θ = 46; present,, Re θ = 46,526. The law log(y + )/.4+5 is the discontinuous straight line in (a). (e) Twodimensional spectral densities from channels (solid) and boundary layers (dashed), present and [8]. [8] J. Jiménez, S. Hoyas, M. P. Simens and Y. Mizuno. (2) Turbulent boundary layers and channels at moderate Reynolds numbers. J. Fluid Mech. 657: