Towards Petascale Computing for Realistic Jet Noise Simulations

Transcription

1 Towards Petascale Computing for Realistic Jet Noise Simulations KURT M. AIKENS NITIN S. DHAMANKAR YINGCHONG SITU Dept. of Comp. Sci. 305 N University St ysitu@cs.purdue.edu YE WANG Dept. of Comp. Sci. 305 N University St wang1351@purdue.edu CHANDRA S. MARTHA cmartha@purdue.edu ZHIYUAN LI Dept. of Comp. Sci. 305 N University St ci@purdue.edu GREGORY A. BLAISDELL blaisdel@purdue.edu ANASTASIOS S. LYRINTZIS Embry Riddle Astro. Univ. Dept. of Aero. Eng. 600 S Clyde Morris Blvd Daytona Beach, FL, USA lyrintzi@erau.edu Abstract: Jet noise is an important issue due to increased commercial air traffic, penalty fees for noisier aircraft, and stringent noise regulations as well as military operational requirements. Advanced computational techniques based on large eddy simulation (LES) have been used to study jet noise. We have previously developed an in-house LES application for simulating subsonic jets without the nozzle. This application is designed to feature supreme scalability, which has enabled large-scale experiments utilizing up to 91,125 processor cores. In this paper, we describe recent advancements of our methodology that allow simulations including nozzles and of jets at supersonic conditions with shock waves. We also discuss future directions in exploring hybrid and heterogeneous parallelism. Keywords: computational fluid dynamics, aeroacoustics, aircraft noise 1 Introduction Aviation has assumed a critical role in supporting global economic growth. Aircraft noise, a byproduct of aviation, however, is proving to have a tangible impact on the overall benefit of aviation, ranging from physical damage to the human body to financial penalties imposed on its originators and costs of noise mitigation measures. In recent years, designing quieter aircraft has become a new arena of competition for major aircraft manufacturers. Low-noise in-flight experience is gaining emphasis in advertising campaigns targeting ordinary customers. In military scenarios, injury-incurring sound levels near vertical and/or short take-off and landing (V/STOL) aircraft also corroborates the necessity of noise reduction in aircraft design. An important design tool of modern reduced-noise aircraft is computational aeroacoustics (CAA) simulation of sound levels generated by aircraft airframes and engines. CAA, a relatively new discipline of computational fluid dynamics (CFD), is intended as a robust and accurate technique that complements traditional theoretical and empirical approaches in aircraft noise prediction. Different from practices in general CFD, CAA strongly relies on accurate prediction of small-amplitude acoustic fluctuations and their correct propagation to the far field. This demands that the underlying numerical methods provide high accuracy and good spectral resolution, while keeping diffusion and dispersion errors low. The state-of-the-art CAA prediction of far-field noise is based on time-dependent CFD simulation of turbulent flows followed by integral postprocessing [18] that propagates the noise to the far field. This two-step approach first appeared in the 1990s and has continued to mature, partly owing to the emergence of large-scale computing platforms, which have turned computationally intensive techniques including direct numerical simulation (DNS) and LES into reality. Reference [1] is a recent review on this subject. We summarize recent high-fidelity simulations below. For subsonic jets, references [35 37] utilize a multiblock solver with overset grid blocks to simulate nozzles with and without chevrons on grids up to 500 million points. They have also performed studies of inflow conditions upstream of the nozzle. In general, their results predict the acoustic high frequencies well ISBN:

2 but underpredict the lower ones. The specific cause of the underprediction is unknown. References [4,5] have studied the effect of important parameters on the noise. Using high-order methods on meshes with 252 million points, they have found that as the Reynolds number and boundary layer turbulence intensities increase, the jet potential core is lengthened, peak turbulence intensities are reduced, and far-field sound pressure levels decrease. This indicates that it is crucial to predict accurate turbulent quantities at the nozzle exit and perform simulations at realistic Reynolds numbers. There are comparatively fewer high-fidelity supersonic jet simulations. While structured solvers for LES have been dominant, there has been a push towards unstructured solvers, which allow for more flexibility in meshing and modeling complex geometries but may also reduce accuracy. Utilizing this approach, [21] includes round converging-diverging nozzles to analyze heated and unheated jets at near-perfect expansion. Other applications include large simulations of hot overexpanded round jets issuing from realistic converging-diverging nozzles with chevrons [6] and underexpanded isothermal rectangular jets from nozzles with and without chevrons [23]. These simulations utilize hundreds of millions of grid points and scale to as many as 163,840 processors on the Intrepid cluster at the Argonne National Laboratory. Additional details on their numerical methods can be found in [10]. The above highlight some of the various LES techniques in use for jet noise studies. A survey of the literature finds that fine grids, high-order numerical methods, and accurate turbulent nozzle boundary layers are required for accurate quantitative sound level predictions. That said, there is much room for improvement and many challenges remain. Examples include simulation of flows at realistically high Reynolds numbers and inclusion of realistic nozzle geometries in high-order structured solvers. We have recently developed a highly-parallel LES application [19, 20] which aims to address these concerns and more accurately predict noise levels at the far field. Our LES application is designed to feature scalability across a wide range of processor core counts and has successfully carried out performance experiments utilizing between 2,744 and 91,125 cores. Previously, its capabilities were limited to simulations without nozzle geometries in subsonic conditions [20]. Recently, we have extended its functionalities for inclusion of nozzle geometries and simulation of supersonic jets with shock waves. We have also explored various options in further improving its performance including hybrid and heterogeneous parallelism. In this paper, we describe the aforementioned efforts in more detail. 2 Methodology of LES In LES, the effect of the large eddies is simulated directly, whereas that of the small eddies is modeled using a subgrid-scale (SGS) model. A variety of SGS models, such as the Smagorinsky [29] and dynamic Smagorinsky [9] models, have been employed in LES of jets. Alternatively, one can use a filter which is part of the numerical method [2, 32] or rely on the dissipation inherent to an upwind biased numerical scheme [25, 26] to provide an implicit SGS model. While unstructured solvers are frequently used for CFD because of their ability to easily handle complex geometries, the accuracy requirements for CAA often call for higher-order numerical schemes. This is due to the small amplitude of the acoustic fluctuations and the wide range of turbulent length scales that must be accurately resolved for many CAA problems. Structured solvers are ideal then, due to their available high-order methodologies and efficiency. In our LES application, the filtered conservativeform Navier Stokes equations are solved in generalized curvilinear coordinates on a uniform grid after spatial transformation. The classical fourth-order Runge Kutta method is used for time integration, and a surface integral method based on the porous surface Ffowcs Williams Hawkings equation [18] is used for evaluating acoustic signals. Spatial partial derivatives appearing in the governing equations are discretized using the following sixth-order compact finite difference scheme from [14]: 1 3 f i 1 + f i f i+1 = 7 9h ( f i+1 f i 1 ) h ( f i+2 f i 2 ) (1) where f i and f i are the values of a function f and its approximate first derivative at the ith point, and h is the grid spacing. For points in the vicinity of the boundaries, lower-order and one-sided implicit difference formulae are used to ensure an overall tridiagonal formulation [14]. For nonlinear problems with nonperiodic boundary conditions and nonuniform grids, the above compact difference scheme is unstable. As a remedy, we use the following sixth-order compact spatial filter from [38] to attenuate the high-wavenumber ISBN:

3 modes: α f f i 1 + f i + α f f i+1 = 3 n=0 a n 2 ( f i n + f i+n ) (2) where f i is the filtered value of f at the ith point, α f is a filter parameter satisfying α f < 0.5, and a 0, a 1, a 2, a 3 are constants depending on α f. As with the compact difference scheme, lower-order and one-sided implicit formulae are used for points in the vicinity of the boundaries. A fast and accurate solver for the diagonally dominant tridiagonal linear systems induced by Equations 1 and 2 is crucial to our LES application. To this end, the transposition [31] and Schur complement methods [15] are known to have severe scalability limitations [20]. Better in this regard are some schemes that have overlapping blocks [33, 41] and constrain the linear systems to individual blocks. This is trivial to parallelize but inherently less accurate. For both high accuracy and scalability, our LES application uses the truncated SPIKE algorithm [24, 27] on non-overlapping blocks to solve these tridiagonal linear systems with theoretically optimal weak-scaling scalability [28]. Reference [20] reports a strong scaling test case with a 1,260 1,260 1,260 grid where our LES application achieves a speedup of 24.6 for an efficiency of 74 % when the number of processor cores increases from 2,744 to 91,125. For natural simulation of jets, advanced support for cylindrical grids has been included in our LES application. Various approaches [3, 22, 34] exist to handle the centerline singularity and avoid the time step penalty associated with the inherent concentrated grid point distribution. We have implemented three approaches: a point skipping method [3], the spectral method, and the windowed sinc filter [19]. The point skipping method produces good results [19], but our implementation uses the transposition method and thus inherits its scalability weaknesses. The sinc filter is more scalable, but some discrepancies have been noticed in the results. This is an area of future research. 3 Recent code implementations 3.1 Inlet and wall boundary conditions To enable the inclusion of nozzle walls in jet simulations, several boundary conditions are implemented in our LES application. First, an adiabatic viscous wall formulation based on characteristic analysis is added [11, 12]. While it works well, its formulation incurs extra computational cost because modifications to individual terms in the inviscid flux derivatives require additional derivative computations. As a result, an extrapolation-based method [16] is added which reduces derivative computations by 33 % compared to the characteristic boundary conditions. With this method, the continuity equation is advanced in time to compute the density on the wall, taking advantage of the no-slip velocity condition. The momentum and energy equations, however, are not time-advanced at the wall. Either an isothermal or adiabatic wall is assumed for calculating the pressure. In test cases of viscous flows over a cylinder, the two formulations are found to agree very well. The extrapolation-based boundary conditions are preferred for their efficiency. In addition to wall boundary conditions, when including the nozzle geometry it is crucial to achieve a fully turbulent boundary layer on the nozzle wall to better match realistic operating conditions with high Reynolds numbers. A realistic turbulent boundary layer is required to accurately feed the turbulent shear layer downstream of the nozzle exit which directly influences the noise. Thus, a digital filter-based turbulent inflow boundary condition [13, 30, 39] is implemented. Furthermore, the method is extended to non-uniform curvilinear coordinates in a novel way which allows limited variation in the specification of integral length scales over the inlet plane [7]. This method has been tested on a spatially evolving zero pressure gradient flat plate turbulent boundary layer in a Reynolds number range of 1,410 Re θ 2,200 based on freestream velocity and momentum thickness. Using the current method, the turbulent stresses recover completely in a distance of about 11.5 boundary layer thicknesses. The velocity profile is also in reasonable agreement with the law of wall, indicating that the method is capable of producing sustained turbulent fluctuations. Figure 1 shows that the turbulence evolves into qualitatively realistic vortex structures in the nozzle boundary layer. The method has since been used successfully in jet simulations. 3.2 Characteristic filters for supersonic jets To extend the implementation to include supersonic simulation capabilities, it is necessary to add shock capturing routines for simulation of off-design jets with shock waves. Based on the success of characteristic filters [8, 40] in our legacy code [17], they are ported to our current LES application. Characteristic filters ISBN:

4 Figure 1: Instantaneous isosurfaces of the Q-criterion inside nozzle with Q = 5(U j /r 0 ) 2. The arrow indicates the flow direction. are based on the idea that discontinuous flow predictions with baseline methodologies can be corrected by adding the dissipative portion of a nonlinear scheme (e.g., total variation diminishing or weighted essentially non-oscillatory schemes) after a full time step. As a result, they are general and efficient since they can be coupled with any nondissipative base numerical scheme and do not need to be applied at each substage of the Runge Kutta method. In our application, they are made even more efficient and less dissipative by applying them locally to shock wave regions only [17]. The shock capturing modules have been validated using simulations without nozzles. Turbulent statistics and acoustics results match well those from the legacy code. Additionally, a centerline treatment module has been developed to reduce CFL-type time step restrictions for cylindrical grids. Analogous to the point-skipping method from [3], the grid is artificially coarsened by adjusting the amount of applied dissipation along rings near the centerline. Supersonic jet acoustics simulations including the nozzle have not yet been completed, but cases with short downstream domains (e.g., 3 jet radii) have been simulated to ensure that the implementation is correct. Figure 2 shows instantaneous contours of velocity magnitude inside a military-style nozzle. Utilizing the turbulent inflow boundary conditions, preliminary results for converging-diverging nozzles are promising. 4 Future directions As processor technologies continue to evolve, computation is outpacing communication in growth of efficiency by a widening margin on current large-scale Figure 2: Wall-resolved simulation of a convergingdiverging nozzle with a design Mach number of Boundary conditions are shown. computing platforms. Specific to our LES application, we previously measured on the Kraken cluster at the National Institute for Computational Sciences (NICS) that communication took approximately a third of the total running time. Our recent experiments on the Carter cluster at the Rosen Center for Advanced Computing (RCAC) of show that that proportion has risen to about 50 %. One possible solution is to adopt multiple forms of parallelism. Currently, our LES application is parallelized using the Message Passing Interface (MPI) and thus exploits the single-program multiple-data (SPMD) parallelism. As we increase the number of MPI processes, the block size decreases, resulting in reduced communication efficiency as [28] shows. To alleviate such an issue, we are investigating running each MPI process on multiple processor cores using OpenMP as the means of intraprocess parallelization. This allows each process to handle larger blocks and enables more flexible load balancing when heterogeneous parallelism is incorporated. Our initial experiments have shown that an MPI OpenMP hybrid implementation can be up to 3 % faster than the MPI-only version. We plan to continue performance tuning for current symmetric multiprocessing (SMP) clusters and prepare for porting the code to Intel Xeon Phi coprocessors. Using graphics processing units (GPUs) is another option for speeding up our LES application. We are currently in the process of experimentally porting the application to NVIDIA GPUs and have finished all truncated SPIKE-related code. Instead of simply offloading workload to the GPU, our strategy is GPUcentric, with the CPU used for control and communi- ISBN:

5 cation only and otherwise left as additional computing power. We have achieved a speedup of 1.5 to 2.5 over the MPI OpenMP version in computation but have seen longer times in communication. The main cause is that in our current implementation, GPU-to-GPU communication has to go through the main memory. We are investigating taking advantage of hardware support for direct GPU-to-GPU communication via the PCI Express bus and remote direct memory access (RDMA) over InfiniBand to reduce communication time and continue to port the entire application to GPUs. We envision that in the long term, our LES application will combine both hybrid parallelism and accelerator-based heterogeneous parallelism. Acknowledgments: This study is primarily supported by the National Science Foundation (NSF) via award CCF , with additional support from award OCI Experiments of this study have used the Kraken cluster hosted by the NICS under Extreme Science and Engineering Discovery Environment (XSEDE) allocation TG-ASC040044N and the Carter cluster of the Rosen Center for Advanced Computing at. XSEDE is supported by NSF award OCI References: [1] D. J. Bodony and S. K. Lele. Current status of jet noise predictions using large-eddy simulation. AIAA Journal, 46(2): , [2] C. Bogey and C. Bailly. LES of a high Reynolds, high subsonic jet: effects of the subgrid modellings on flow and noise. AIAA Paper , AIAA, [3] C. Bogey, N. de Cacqueray, and C. Bailly. Finite differences for coarse azimuthal discretization and for reduction of effective resolution near origin of cylindrical flow equations. Journal of Computational Physics, 230(4): , [4] C. Bogey, O. Marsden, and C. Bailly. Effects of moderate reynolds numbers on subsonic round jets with highly disturbed nozzle-exit boundary layers. Physics of Fluids, 24: , [5] C. Bogey, O. Marsden, and C. Bailly. Influence of initial turbulence level on the flow and sound fields of a subsonic jet at a diameter-based Reynolds number of Journal of Fluid Mechanics, 701: , [6] G. A. Brès, Joseph W Nichols, S. K. Lele, Frank E. Ham, R. H. Schlinker, R. A. Reba, and J. C. Simonich. Unstructured large eddy simulation of a hot supersonic over-expanded jet with chevrons. AIAA Paper , AIAA, [7] N. S. Dhamankar. Boundary conditions towards realistic simulation of jet engine noise. Master s thesis, School of Aeronautics and Astronautics,, [8] E. Garnier, P. Sagaut, and M. Deville. A class of explicit ENO filters with applications to unsteady flows. Journal of Computational Physics, 170(1): , [9] M. Germano, U. Piomelli, P. Moin, and W. H. Cabot. A dynamic subgrid-scale eddy viscosity model. Physics of Fluids A: Fluid Dynamics, 3(7): , [10] Y. Khalighi, J. W. Nichols, S. K. Lele, F. E. Ham, and P. Moin. Unstructured large eddy simulations for prediction of noise issued from turbulent jets in various configurations. AIAA Paper , AIAA, [11] J. W. Kim and D. J. Lee. Generalized characteristic boundary conditions for computational aeroacoustics. AIAA Journal, 38(11): , [12] J. W. Kim and D. J. Lee. Generalized characteristic boundary conditions for computational aeroacoustics, part 2. AIAA Journal, 42(1):47 55, [13] M. Klein, A. Sadiki, and J. Janicka. A digital filter based generation of inflow data for spatially developing direct numerical or large eddy simulations. Journal of Computational Physics, 186(2): , [14] S. K. Lele. Compact finite difference schemes with spectral-like resolution. Journal of Computational Physics, 103(1):16 42, [15] P.-T. Lew. A study of sound generation from turbulent heated round jets using 3-D large eddy simulation. PhD thesis, School of Aeronautics and Astronautics,, [16] J. Liu, C. Kaplan, and E. Oran. A brief note on implementing boundary conditions at a solid wall using the FCT algorithm. Memorandum report NRL/MR/ , Naval Research Laboratory, [17] S.-C. Lo, K. M. Aikens, G. A. Blaisdell, and A. S. Lyrintzis. Numerical investigation of 3- D supersonic jet flows using large-eddy simulation. International Journal of Aeroacoustics, 11(7): , ISBN:

6 [18] A. S. Lyrintzis. Surface integral methods in computational aeroacoustics from the (CFD) nearfield to the (acoustic) far-field. International Journal of Aeroacoustics, 2(2):95 128, [19] C. S. Martha. Toward high-fidelity subsonic jet noise prediction using petascale supercomputers. PhD thesis, School of Aeronautics and Astronautics,, [20] C. S. Martha, Y. Situ, M. E. Louis, G. A. Blaisdell, A. S. Lyrintzis, and Z. Li. Development and application of an efficient, multiblock 3-D large eddy simulation tool for jet noise. AIAA Paper , AIAA, [21] S. Mendez, M. Shoeybi, A. Sharma, F. E. Ham, S. K. Lele, and P. Moin. Large-eddy simulations of perfectly expanded supersonic jets using an unstructured solver. AIAA Journal, 50(5): , [22] K. Mohseni and T. Colonius. Numerical treatment of polar coordinate singularities. Journal of Computational Physics, 157(2): , [23] J. W. Nichols, S. K. Lele, P. Moin, F. E. Ham, and J. E. Bridges. Large-eddy simulation for supersonic rectangular jet noise prediction: effects of chevrons. AIAA Paper , AIAA, [24] E. Polizzi and A. H. Sameh. A parallel hybrid banded system solver: the SPIKE algorithm. Parallel Computing, 32(2): , [25] M. L. Shur, P. R. Spalart, and M. K. Strelets. Noise prediction for increasingly complex jets. Part I: Methods and tests. International Journal of Aeroacoustics, 4(3 4): , [26] M. L. Shur, P. R. Spalart, and M. K. Strelets. Noise prediction for increasingly complex jets. Part II: Applications. International Journal of Aeroacoustics, 4(3 4): , [27] Y. Situ, L. Liu, C. S. Martha, M. E. Louis, Z. Li, A. H. Sameh, G. A. Blaisdell, and A. S. Lyrintzis. Reducing communication overhead in large eddy simulation of jet engine noise. In 2010 IEEE International Conference on Cluster Computing, pages , [28] Y. Situ, L. Liu, C. S. Martha, M. E. Louis, Z. Li, A. H. Sameh, G. A. Blasidell, and A. S. Lyrintzis. A communication-efficient linear system solver for large eddy simulation of jet engine noise. Cluster Computing, 16(1): , [29] J. Smagorinsky. General circulation experiments with the primitive equations. Monthly Weather Review, 91(3):99 164, [30] E. Touber and N. D. Sandham. Large-eddy simulation of low-frequency unsteadiness in a turbulent shock-induced separation bubble. Theoretical and Computational Fluid Dynamics, 23(2):79 107, [31] A. Uzun, G. A. Blaisdell, and A. S. Lyrintzis. Application of compact schemes to large eddy simulation of turbulent jets. Journal of Scientific Computing, 21(3): , [32] A. Uzun, G. A. Blaisdell, and A. S. Lyrintzis. Impact of subgrid-scale models on jet turbulence and noise. AIAA Journal, 44(6): , [33] A. Uzun and M. Y. Hussaini. High-fidelity numerical simulations of a chevron nozzle jet flow. AIAA Paper , AIAA, [34] A. Uzun and M. Y. Hussaini. Simulation of noise generation in near-nozzle region of a chevron nozzle jet. AIAA Journal, 47: , [35] A. Uzun and M. Y. Hussaini. High-fidelity numerical simulations of a round nozzle jet flow. AIAA Paper , AIAA, [36] A. Uzun and M. Y. Hussaini. On some issues in large-eddy simulations for chevron nozzle jet flows. AIAA Paper , AIAA, [37] A. Uzun and M. Y. Hussaini. Some issues in large-eddy simulations for chevron nozzle jet flows. Journal of Propulsion and Power, 28(2): , [38] M. R. Visbal and D. V. Gaitonde. Very highorder spatially implicit schemes for computational acoustics on curvilinear meshes. Journal of Computational Acoustics, 9(4): , [39] Z.-T. Xie and I. P. Castro. Efficient generation of inflow conditions for large eddy simulation of street-scale flows. Flow, turbulence and combustion, 81(3): , [40] H. C. Yee, N. D. Sandham, and M. J. Djomehri. Low-dissipative high-order shockcapturing methods using characteristic-based filters. Journal of Computational Physics, 150: , [41] X. Zhang, G. A. Blaisdell, and A. S. Lyrintzis. High-order compact schemes with filters on multi-block domains. Journal of Scientific Computing, 21: , ISBN: