OpenFOAM Parallel Performance on HPC

Transcription

1 OpenFOAM Parallel Performance on HPC Chua Kie Hian (Dept. of Civil & Environmental Engineering) 1 Introduction OpenFOAM is an open-source C++ class library used for computational continuum mechanics simulations. While OpenFOAM is mainly used for Computational Fluid Dynamics (CFD) simulations, it can be applied to other simulations such as solid dynamics and the pricing of financial options. OpenFOAM has seen increasingly rapid development since its creation in This is can be attributed to factors such as the availability of the source-code and reduction in cost of parallel computing. Parallel computing hardware is now easily available as workstations, computing clusters or via rental from a service such as Amazon Elastic Compute Cloud. 2 Scope The objective of this article is to investigate the parallel performance of OpenFOAM on the NUS HPC cluster. Two sample cases were tested whereby each case was ran with 1, 2, 4, 8, 12 and 16 CPUs. Comparisons of memory (RAM and Swap) usage and computation (both CPU and clock) times were made for each test run. It should be noted that in general, simulations with small mesh sizes will not see significant speedups in parallelisation. This is because the computation time for small meshes may be relatively short compared to the amount time the processor units spend for communication and message exchange. Therefore, in order to make the comparisons more meaningful, the mesh sizes of the sample cases were increased in the test runs. 3 Results 3.1 Case 1 - Dam Break The dam break case, available under the tutorials/multiphase/interfoam/ras subdirectory of a standard OpenFOAM installation, was used in the first set of test runs. The case simulated a wall of water collapsing onto an obstacle and illustrated OpenFOAM s ability to simulate complex multiphase flows. The screen captures below show the various stages of the simulation.

2 (a) Dam break: t=0 sec (b) Dam break t=0.15 sec (c) Dam break: t=0.30 sec (d) Dam break: t=0.50 sec Figure 1: Dam break simulation For the test runs, the mesh size was increased to 25,684 cells. Figure 2 presents the results of the test runs, comparing the computational time. The CPU time refers to the actual amount of time spent on computation while the clock time records the total time taken to complete the run - the difference between the two time values represents the time spent on inter-core message exchange. Figure 2: Dam break case: computational time

3 Comparing the computational time, it can be seen that speedups can be gained from parallelisation, up to 8 CPUs. Beyond that, the time taken for inter-cpu message exchange becomes greater than the CPU computation time, hence resulting in reduced speedups compared to a serial (single CPU) case. Figure 3 shows the memory usage of the dam break test runs. It can be seen that both the RAM and swap increased linearly with the number of CPUs used. Figure 3: Dam break case: memory usage 3.2 Case 2 - Solitary Wave The solitary wave case was taken from a wave generation and absorption toolbox developed at the Technical University of Denmark 1. (a) (b) (c) (d) (a) Solitary Wave: start (wave generated at left boundary) (b) Solitary Wave: after 10 seconds (c) Solitary Wave: after 20 seconds (d) Solitary Wave: after 22.5 seconds (wave absorbed at right boundary) Figure 4: Solitary wave simulation 1 Reference: Jacobsen, N. G., Fuhrman, D. R., Fredsøe, J., A wave generation toolbox for the open-source CFD library: OpenFOAM ;, International Journal for Numerical Methods in Fluids, 2011

4 The case simulated the propagation of a solitary wave across a 2-D computational domain and demonstrated wave generation and absorption capability, in addition to two-phase flow modelling. The screen captures below show the various stages of the simulation. A mesh size of 59,995 cells was used for the test runs. Figure 5 presents the comparison of computational times for the various runs of the solitary wave case. The observation was similar to that made for the dam break case, ie parallelisation of the simulation reduced computational times, up to a limit of 8 CPUs. Beyond that, the time taken for the computation actually increased due to the high overhead of data traffic between the CPUs. This can be observed from the fact that although the CPU time continued to decrease for parallelisation beyond 8 CPUs, the clock time actually increased. Figure 5: Solitary wave case: computational time Figure 6 shows the memory usage for the various runs of the solitary wave case. The observation here is similar to that of the dam break case. Although it can be said that computational efficiency gains from parallelisation diminished beyond 8 cores, running a parallel case across 12 or 16 cores still sped up the simulation by about 6 times.

5 Figure 6: Solitary wave case: memory usage 4 Concluding Remarks The investigations indicate that parallelisation of a simulation can result in performance gains. However, the extent of speedup depends on the size and complexity of the simulation - parallelisation of relatively small simulations (in terms of mesh size) may not give significant performance gains due to the inter-cpu communication overhead loads. Generally, when conducting research, a range of cases may be investigated by varying certain parameters. This further increases the complexity of a simulation, making it harder to judge the optimal extent of parallelisation. Furthermore, in a load-sharing cluster scenario, the queue priority of a cluster setup also plays a part when deciding the parallelisation strategy - while decomposing the simulation across a large number of domains may theoretically speed it up, the decomposed run may end up spending a large amount of time in the queue as the cluster needs to wait until a sufficient number of cores is available to process the job.