Large Data Set Computations with CFX: To 100 Million Nodes & Beyond!

Transcription

1 Large Data Set Computations with CFX: To 100 Million Nodes & Beyond! Dr. Mark E. Braaten Mechanical Engineer GE Global Research, Niskayuna, NY

2 Acknowledgements Thanks to many people, in particular: GEGR Colleagues Stuart Connell, Semir Kapetanovic, Adam Stevenson, Peter Schmid CFX Colleagues Mike Marchant, Andy Mortimer, David Main (UK) Dan Williams, Phil Zwart, Steve Elias, Paul Galpin, Hrvoje Roglic (Canada)

3 Large Data Set Project Goal of Large Data Set Project is to look at hardware / software issues that arise when running very large CFD applications (>100M nodes ) CFX and GE in-house turbomachinery code Examine scalability of current CFX system to very large problems Look at mesh generation, pre-processing, partitioning, solver, postprocessor Create series of cases of increasing size Roughly 1M, 2M, 10M, 25M, 50M, 100M nodes Run on 64-bit desktop and 32/64-bit Linux clusters Expect these to be the likely computer resources available in the near future

4 Current Status of CFX-10.0 CFX can handle both unstructured and structured meshes CFX-Solver has demonstrated parallel scalability on Linux clusters (32- and 64-bit) CFX-Pre, CFX-Post, and the CFX Partitioner are available for 64-bit machines CFX-Mesh is currently limited to 32-bit, due to its integration into ANSYS Workbench Impossible to make single mesh larger than about 5M nodes CFX Partitioner is serial only Serial METIS has a severe memory bottleneck

5 Selection of Large CFX Test Case IDEA: Use an existing block-structured mesh for a turbomachinery passage, and replicate it to make larger problems Advantages: Easy to make sequence of larger problems Avoids current problem with 32-bit mesh generation Models a problem of interest to Large Data Set project Multiple identical passages, multiple dissimilar passages (AVP, MPT), single large mesh Easy to assess solutions on multiple passages Should be same as a single passage replicated Allows use of simple circumferential partitioning Avoids METIS memory requirements

6 CFX Test Case Use typical turbine blade as the test case Convenient for generating series of test cases Mesh for one passage is about 1 M nodes Full annulus contains 92 passages (88M nodes) Block-structured mesh (95 blocks, 970,000 grid points)

7 Large Data Set CFX Test Case (cont d) One passage base case Block-structured mesh (95 blocks) imported directly into CFX-Pre 970,000 grid points per passage Replicate passages to generate larger and larger test cases 2 passages ~ 2 M nodes 10 passages ~ 10 M nodes 23 passages (quarter annulus) ~ 22 M nodes 46 passages (half annulus) ~ 44 M nodes 92 passages (full annulus) ~ 88 M nodes

8 Computer Resources Initially CFX-Pre, CFX-Partition, and CFX- Post run on 64-bit Itanium machine Process has since been repeated using new EM64T- based ATW machines that are dualbooted with 64-bit Linux O.S. CFX-Solver run on Linux clusters at GEGR 32-bit cluster with 2GB memory per processor 64-bit cluster with 3GB memory per processor

9 CFX-Pre Had existing problem setup for a single passage Exported CCL from this case Process: Import single passage grid Replicate grid Specify number of copies, angle/axis of rotation Import CCL from single passage case Point boundary conditions to replicated faces Takes less than 15 minutes (start to finish) for even full annulus case No significant scalability issues for replicated passages Memory increases linearly for independent passages Can reach almost 200M node hex mesh w/ 16GB memory Not a barrier for now

10 Ten Passages

11 Twenty Three Passages (Quarter Annulus)

12 Ninety Two Passages (Full Annulus)

13 Full Annulus Simulation in CFX passages 88M grid points Hub cavity inlet inlet exit Casing cavity inlet

14 CFX-Partitioner Used simple circumferential partitioning to divide up the final mesh Avoids memory requirements of METIS Partitioning is reasonable for problem of this type Set number of processors ~equal to number of passages One million nodes per processor 1.5GB memory per processor within 32-bit limit Cases range from 1 to 92 processors Required >9 GB memory for 92 passage case (i.e. a 64-bit machine with lots of memory) Need a parallel METIS partitioner for a more general problem To be discussed later

15 Partitioner Run Partitioning Information Partitioning of domain: Domain 1 Partitioner run on Itanium- 23 processors 9.1 GB memory 30 minutes cpu - Partitioning tool: Cirumferential direction (weighted) - Number of partitions: 23 - Number of nodes: Partitioning axis from: ( 0.000E+00, 0.000E+00, 0.000E+00) - Partitioning axis to: ( 0.000E+00, 0.000E+00, 1.000E+00) Partitioning information for domain: Domain Elements Vertices (Overlap) Faces Weight Full mesh Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Part % Sum of part % CPU-Time requirements: - Preparations 3.182E-01 seconds - Low-level mesh partitioning 7.375E+02 seconds - Global partitioning information 1.675E+02 seconds - Vertex, element and face partitioning information 5.516E+02 seconds - Element and face set partitioning information 4.840E+01 seconds - Summed CPU-time for mesh partitioning 1.754E+03 seconds

16 Partitioned Mesh 92 partitions

17 CFX Solutions The following slides show some typical results Results with different number of passages agree very well Parallel scalability is very good Converged solution for full annulus (88M nodes) obtained in less than 8 hours clock time

18 Ten Passages (10M nodes)

19 Forty-Six Passages (44M nodes) Half annulus

20 Ninety-Two Passages (88M nodes) 7 hr, 47 min clock time w/ 90 processors, 100 iterations Full annulus According to CFX, this is the largest realistic computation to date w/ CFX

21 Convergence Path

22 File Sizes, Run Times #Pass. Mesh Size Definition Results Clk Time M 20 MB 300MB 7.22 hrs 2 1.9M 46 MB 618 MB 5.75 hrs M 255 MB 2958 MB 23 22M 597 MB 6822 MB 46 44M 1200 MB MB M 2407 MB 9960 MB hrs Footnotes: 1 Timings on 64-bit Linux cluster 2 90 processors 3 Full results files (the default) contain many unnecessary variables. Choosing Selected Variables makes results files only 1/3 as large

23 Memory and I/O Memory usage 1.2 GB memory allocated per processor 300 words per node allocated Actual usage closer to 250 words per node CFX-10.0 does all I/O on the master processor Serial I/O paradigm I/O times (writing to project share) 1 processor < 1 minute for final file write 90 processors ~ 45 minutes for final file write File compression runs serially on master processor Slows file output significantly File compression will be done in parallel in CFX-11.0

24 Beyond 100M nodes Very large block-structured meshes for single blade passage recently obtained from GE colleagues (Tolpadi, Sewall) Mesh 1: 3.1 million nodes Mesh 2: 5.1 million nodes Replicated 64 passages to create full annulus in CFX-Pre Mesh M nodes (5.5 GB definition file) Mesh M nodes (9.9 GB definition file)

25 Beyond 100M nodes (cont d) Ran into limitation in CFX-10.0 writing definition files Reported problem to CFX Development CFX determined cause of problem, developed fix within two weeks Received preview version of CFX-11.0 that corrects problem

26 The Largest CFX Mesh Yet Linux!

27 327 Million Node Mesh Rendered in CFX-POST Memory Requirements: PostEngine: 23GB PostGui 5GB wireframe

28 327 Million Node Mesh (cont d) Mesh

29 Biggest CFX Needs for Large Data Set Cases To date, large data set project has identified two major needs for CFX bit versions of ANSYS Workbench and CFX- Mesh For unstructured meshing of large single grids 2. Parallel METIS partitioner

30 Biggest CFX Needs for Large Data Set Cases (cont d) Projects to develop these are underway CFX-Mesh has been compiled using Intel 8 compiler on 64-bit Itanium Included in Service Pack 1 for Workbench 10.0 on XP bit Linux version under development for Workbench Parallel METIS has been demonstrated for CFX unstructured meshes, block structured meshes as part of this project See following charts

31 Parallel METIS Prototype Current Status A working prototype has been developed to use parallel METIS to partition a CFX definition file Three components of prototype: 1. Convert CFX definition file to ASCII input file for ParMETIS 2. Run ParMETIS for mixed mesh of tetrahedra, pyramids, prisms, hexes 3. Convert output from ParMETIS to CFX partition file Meshes up to 88M nodes have been partitioned with excellent results Actual partitioning of 88M node mesh into 92 partitions took only 40 cpu sec on 46 processors

32 Parallel Partitioning of a CFX Mesh using PARMetis 1.2 M nodes, 16 partitions, 16 processors 0 Partition % Element % Node E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E+00

33 Another Example of Parallel Partitioning of a CFX Mesh 44 M nodes, 46 partitions, 46 processors Partitioning itself takes only 20 cpu seconds!

34 Concluding Remarks CFX-10.0 has demonstrated the ability to run problems exceeding 100 million grid points on present hardware Converged solution obtained for complete turbine wheel in under 8 hours on 90 processors Solver can run well on both 32-bit and 64-bit Linux clusters, with good parallel scalability A large memory (16GB) 64-bit OS machine is needed for pre- and post-processing of large cases Billion node CFX computations are not far off Come back next year!