28.1 Introduction to Parallel Processing

Transcription

1 Chapter 28. Parallel Processing The FLUENT solver allows for parallel processing and provides tools for checking and modifying the parallel configuration. You can use a dedicated parallel machine (for example, a multiprocessor workstation), or you can run your solver on a network of workstations. The following sections describe the parallel-processing features of FLUENT. Section 28.1: Introduction to Parallel Processing Section 28.2: Starting the Parallel Version of the Solver Section 28.3: Using a Parallel Network of Workstations Section 28.4: Partitioning the Grid Section 28.5: Checking and Improving Parallel Performance 28.1 Introduction to Parallel Processing The parallel version of FLUENT is a different version of the solver that simultaneously computes the solution using multiple compute nodes (processes). To use it, you will partition your grid into multiple subdomains such that the number of partitions is an integral multiple of the number of compute nodes available to you (e.g., 8 partitions for 1, 2, 4, or 8 compute nodes). Each partition (or agglomerated group of partitions) will then reside on a different compute node. These processes can be compute nodes on a massively parallel computer, processes on a multiple- CPU workstation, or processes on a network of heterogeneous workstations running UNIX, or on a network of workstations running Windows. In general, as the number of compute nodes increases, turnaround time for the solution will decrease. However, parallel efficiency decreases as the ratio of communication to computation increases, so you should be careful to choose a large enough problem for the parallel machine. c Fluent Inc. November 28,

2 Parallel Processing The recommended procedure for using parallel FLUENT is as follows: 1. Start up the parallel solver and spawn additional compute nodes (if necessary). See Sections 28.2 and 28.3 for details. 2. Read your case file and have FLUENT partition the grid automatically upon loading it. It is best to partition after the problem is set up, since partitioning has some model dependencies (e.g., adaption on non-conformal interfaces, sliding-mesh and shell-conduction encapsulation). If your case file contains sliding meshes, or nonconformal interfaces on which you plan to perform adaption during the calculation, you will have to partition it in the serial solver. Note that there are other approaches for partitioning, including manual partitioning in either the serial or the parallel solver. See Section 28.4 for details. 3. Review the partitions and perform partitioning again, if necessary. See Section for details on checking your partitions. 4. Calculate a solution. See Section 28.5 for information on checking and improving the parallel performance Starting the Parallel Version of the Solver The way you start the parallel version of FLUENT depends on whether you are using a dedicated parallel machine or a workstation cluster Starting the Parallel Solver on a UNIX System You can run FLUENT on a UNIX dedicated parallel machine or a network of UNIX workstations. The procedures for starting these versions are described in this section. Running on a Multiprocessor UNIX Machine To run FLUENT on a dedicated parallel machine (i.e., a multiprocessor workstation or a massively parallel machine), type the usual startup command without a version (i.e., fluent), and then use the Select Solver 28-2 c Fluent Inc. November 28, 2001

3 28.2 Starting the Parallel Version of the Solver panel (Figure ) to specify the parallel architecture and version information. File Run... Figure : The Select Solver Panel 1. Under Versions, specify the 3D or 2D single- or double-precision version by turning the 3D and Double Precision options on or off, andturnontheparallel option. 2. Under Options, select the message-passing library in the Communicator drop-down list. The Default library is recommended, because it selects the library that should provide the best overall parallel performance for your dedicated parallel machine. c Fluent Inc. November 28,

4 Parallel Processing If you prefer to select a specific library, you can choose either Vendor MPI or Shared Memory MPI (MPICH). Vendor MPI selects the message-passing library optimized by your hardware vendor. If the parallel toolkit supplied by your hardware vendor is installed on your machine, FLUENT will detect it automatically when the Default option is selected. Shared Memory MPI (MPICH) selects the MPICH message-passing library, a public-domain version of MPI. 3. Set the number of CPUs in the Processes field. 4. Click on the Run button to start the parallel version. No additional setup is required once the solver starts. If you prefer to start the parallel version from the command line, you can type fluent version -tn [-pcomm] [-loadhost] [-pathpath]! where version is 2d, 3d, 2ddp, or3ddp, andn is replaced by the number of CPUs to be used. The remaining arguments are optional, as indicated by the square brackets around them. (If you enter one or more of these optional arguments, do not include the square brackets.) comm is replaced by the name of the parallel communication library, host is replaced by the hostname of the machine to launch the compute nodes (by default, it is set to the machine you re on when entering this command line), and path is replaced by the root path to the Fluent.Inc installation directory. In general, you will need to specify -pcomm only if you want to override the default communication library (which should provide best overall parallel performance). The available communicators for dedicated parallel machines and the associated communication libraries for them are listed below: vmpi smpi net vendor MPI shared memory MPI (MPICH) socket See step 2, above, for a description of these libraries c Fluent Inc. November 28, 2001

5 28.2 Starting the Parallel Version of the Solver Running on a UNIX Workstation Cluster To run FLUENT on a network of UNIX workstations, type the usual startup command without a version (i.e., fluent), and then use the Select Solver panel (Figure ) to specify the parallel architecture and version information. File Run Under Versions, specify the 3D or 2D single- or double-precision version by turning the 3D and Double Precision options on or off, andturnontheparallel option.! 2. Under Options, select the Socket message-passing library in the Communicator drop-down list. When you start the parallel network version, you must select Socket or Network MPI (MPICH) in the Communicator drop-down list, unless the vendor MPI library (described earlier in this section) supports clustering. If you keep the Default option, one of the MPI parallel versions will start instead, and you will be unable to spawn additional compute nodes. 3. Set the number of initial compute node processes to spawn on the host machine in the Processes field. You can start with 1 or 0 nodes and spawn the rest later on, as described in Section (optional) Specify the name of a file containing a list of machines, one per line, in the Hosts File field. If the number of Processes is set to 0, FLUENT will spawn a compute node on each machine listed in the file. 5. Click on the Run button to start the parallel network version. If you prefer to start the parallel network version from the command line, you can type fluent version -t1 -pnet (to use the socket communicator) or c Fluent Inc. November 28,

6 Parallel Processing fluent version -t1 -pnmpi (to use the network MPI communicator) to start the solver with 1 compute node on the host workstation. You can then spawn additional processes on remote workstations using the Network Configuration panel, as described in Section You can type fluent version -t0 -pnet [-cnf=hostsfile] (to use the socket communicator) or fluent version -t0 -pnmpi [-cnf=hostsfile] (to use the network MPI communicator) to start a host process that controls compute nodes situated on remote machines. If the optional -cnf=hostsfile is specified, a compute node will be spawned on each machine listed in the file hostsfile. (If you enter this optional argument, do not include the square brackets.) Otherwise, you can spawn the processes as described in Section Starting the Parallel Solver on a Windows System You can run FLUENT on a Windows dedicated parallel machine or a network of Windows machines. The procedures for starting these versions are described in this section. Running on a Multiprocessor Windows Machine On a Windows system, you can start the dedicated parallel version of FLUENT from the MS-DOS Command Prompt window. To start the parallel version on x processors, type fluent version -tx at the prompt, replacing version with the solver version (2d, 3d, 2ddp, or 3ddp) andx with the number of processors (e.g., fluent 3d -t3 to run the 3D version on 3 processors). (See Section for information about modifying your user environment if the fluent command is not recognized.) 28-6 c Fluent Inc. November 28, 2001

7 28.2 Starting the Parallel Version of the Solver Running on a Windows Cluster There are two ways to run FLUENT in parallel on a network of Windows machines: using the RSHD communicator software that is included with the FLUENT distribution, or using a vendor-supplied message-passing interface (VMPI). See the installation instructions for Windows parallel for details about obtaining and installing one of these programs. The startup instructions below assume that you have properly set up the necessary software, based on the appropriate installation instructions. Starting the RSHD-Based Parallel Version of FLUENT If you are using the RSHD software for network communication, type the following in an MS-DOS Command Prompt window: fluent version -pnet [-pathsharename] [-cnf=hostfile] -tnprocs where version must be replaced by the version of FLUENT you want to run (2d, 3d, 2ddp, or3ddp). -pathsharename specifies the shared network name for the Fluent.Inc directory in UNC form. This input is required only if you are not starting the job on the same computer on which FLUENT has been installed. (If you do include this input, do not include the square brackets; see the example below.) If you start the job on the same computer, you do not need to enter this information. For example, if FLUENT has been installed on computer1, then you should replace sharename by the UNC name for the shared directory, \\computer1\fluent.inc. -cnf=hostfile (optional) specifies the hostfile, which contains a list of the computers on which you want to run the parallel job. If the hostfile is not located in the directory where you are typing the startup command, you will need to supply the full pathname to the file. (If you include the -cnf option, do not include the square brackets; see the example below.) c Fluent Inc. November 28,

8 Parallel Processing You can use a plain text editor like Notepad to create the hostfile. The only restriction on the filename is that there should be no spaces in it. For example, hosts.txt is an acceptable hostfile name, but my hosts.txt is not. Your hostfile (e.g., hosts.txt) might contain the following entries: computer1 computer2! The first computer in the list must be the name of the local computer you are working on. If a computer in the network is a multiprocessor, you can list it more than once. For example, if computer1 has 2 CPUs, then, to take advantage of both CPUs, the hosts.txt file should list computer1 twice: computer1 computer1 computer2 If you do not include the -cnf option, FLUENT will start nprocs (see below) processes on the computer where you type the startup command. You can then use the Network Configuration panel in FLUENT to interactively spawn additional nodes on the cluster. See Section 28.3 for details. -tnprocs specifies the number of processes to use. If the -cnf option is present, the hostfile argument is used to determine which computers to use for the parallel job. For example, if there are 10 computers listed in the hostfile and you want to run a job with 5 processes, set nprocs to 5 (i.e., -t5) andfluent will use the first 5 machines listed in the hostfile. You can use the Network Configuration panel to kill processes or spawn additional processes after startup. See Section 28.3 for details c Fluent Inc. November 28, 2001

9 28.2 Starting the Parallel Version of the Solver As an example, the full command line to start a 3D RSHD-based parallel job on the first 3 computers listed in a hostfile called hosts.txt is as follows: fluent 3d -pnet -cnf=hosts.txt -path\\computer1\fluent.inc -t3 Starting the Vendor-MPI-Based Parallel Version of FLUENT If you are using the vendor-supplied MPI software for network communication, type the following in an MS-DOS Command Prompt window: fluent version -pvmpi [-pathsharename] -cnf=hostfile -tnprocs where the options have the same meanings as for the RSHD-based startup described above, with the following differences: The hostfile specification is required. You cannot spawn nodes on the cluster using the Network Configuration panel when this MPI software is used. (Recall that, as for the RSHD-based version, the first computer listed in the hostfile must be the name of the local computer you are working on.) You cannot use the Network Configuration panel to kill processes or spawn additional processes after startup when this MPI software is used. As an example, the full command line to start a 3D vendor-mpi-based parallel job on the first 3 computers listed in a hostfile called hosts.txt is as follows: fluent 3d -pvmpi -cnf=hosts.txt -path\\computer1\fluent.inc -t3 c Fluent Inc. November 28,

10 Parallel Processing 28.3 Using a Parallel Network of Workstations You can create a virtual parallel machine by spawning (and killing) compute node processes on workstations connected by a network. Multiple compute node processes are allowed to exist on the same workstation, even if the workstation contains only a single CPU Configuring the Network If you want to spawn compute nodes on several different machines, or if you want to make any changes to the current network configuration (e.g., if you accidentally spawned too many compute nodes on the host machine when you started the solver), you can use the Network Configuration panel (Figure ). Parallel Network Configure... Figure : The Network Configuration Panel c Fluent Inc. November 28, 2001

11 28.3 Using a Parallel Network of Workstations Structure of the Network Compute nodes are labeled sequentially starting at 0. In addition to the compute node processes, there is one host process. The host process is automatically started when FLUENT starts, and it is killed when FLU- ENT exits. It cannot be killed while running. Compute nodes, however, can be killed at any time, with the exception that compute node 0 can only be killed if it is the last remaining compute node process. The host process always spawns compute node 0. Compute node 0 spawns all other compute nodes. Steps for Spawning Compute Nodes The basic steps for spawning compute nodes are as follows: 1. Choose the host machine(s) on which to spawn compute nodes in the Available Hosts list. If the desired machine is not listed, you can use the Host Entry fields to manually add a host (as described below), or you can copy the desired host from the host database (as described in Section ). 2. Set the number of compute node processes to spawn on each selected host machine in the Spawn Count field. 3. Click on the Spawn button and the new node(s) will be spawned and added to the Spawned Compute Nodes list. Additional functions related to network configuration are described below. Adding Hosts Manually To add a host to the Available Hosts list in the Network Configuration panel manually, you can enter the internet name of the remote machine in the Hostname field under Host Entry, enter your login name on that machine in the Username field (unless your accounts all have the same login name, in which case you need not specify a username), and then click on the Add button. The specified host will be added to the Available Hosts list. c Fluent Inc. November 28,

12 Parallel Processing Deleting Hosts To delete a host from the Available Hosts list in the Network Configuration panel, select the host and click on the Delete button. The host name will be removed from the Available Hosts list (but the hosts database (see Section ) will not be affected). Killing Compute Nodes! If you spawn an undesired compute node, you can easily remove it by selecting it in the Spawned Compute Nodes list and clicking on the Kill button. Remember that compute node 0 can only be killed if it is the last remaining compute node process. Saving a Hosts File If you have compiled a group of Available Hosts that you may want to use again in another session, you can save a hosts file containing all entries in the Available Hosts list. Click on the Save... button and, in the resulting Select File dialog box, enter the name of the file and save it. In a future session, you can load the contents of this file into the hosts database (see Section ) and then copy the hosts over to the Network Configuration panel in order to reproduce the current Available Hosts list. Common Problems Encountered During Node Spawning The spawning process will try to establish a connection with a new compute node, but if after 50 seconds it receives no response from the new compute node, it will assume the spawn was unsuccessful. The spawn will be unsuccessful, for example, if the remote machine is unable to find the FLUENT executable. To manually test if the spawning machine can start a new compute node, you can type rsh [-l username] hostname fluent -t0 -v from a shell prompt on the spawning machine. hostname should be replaced with the internet name of the machine on which you want to c Fluent Inc. November 28, 2001

13 28.3 Using a Parallel Network of Workstations! spawn a compute node, and username should be replaced with your login name on the remote machine specified by hostname. If all your accounts have the same login name, you do not need to specify a username. (The square brackets around -l username indicate that it is not always required; if you do enter a login name, do not include the square brackets.) Note that on some systems, the remote shell command is remsh instead of rsh. The spawn test could fail for several reasons: Login incorrect. The machine spawning a new compute node must be able to rsh to the machine where the new process will reside, or the spawn will fail. There are several ways to enable this capability. Consult your systems administrator for assistance. fluent: Command not found. The rsh to the remote machine succeeded, but the path to the FLUENT shell script could not be found on that machine. If you are using csh, then the path to the FLUENT shell script should be added to the path variable in your.cshrc file. If that also fails, you can use the parallel/network/ path text command to set the path to the Fluent.Inc installation directory directly before spawning the compute node. parallel network path The Hosts Database When you are creating a parallel network of workstations, it is convenient to start with a list of machines that are part of your local network (a hosts file ). You can load a file containing these names into the hosts database and then select the hosts that are available for creating a parallel configuration (or network) on a cluster of workstations using the Hosts Database panel (Figure ). Parallel Network Database... (You can also open this panel by clicking on the Database... button in the Network Configuration panel.) c Fluent Inc. November 28,

14 Parallel Processing Figure : The Hosts Database Panel c Fluent Inc. November 28, 2001

15 28.3 Using a Parallel Network of Workstations If the hosts file fluent.hosts or.fluent.hosts exists in your home directory, its contents are automatically added to the hosts database at startup. Otherwise, the hosts database will be empty until you read in ahostfile. Reading Hosts Files If you have a hosts file containing a list of machines on your local network, you can load this file into the Hosts Database panel by clicking on the Load... button and specifying the file name in the resulting Select File dialog box. Once the contents of the file have been read, the host names will appear in the Hosts list. (FLUENT will automatically add the IP (Internet Protocol) address for each recognized machine. If a machine is not currently on the local network, it will be labeled unknown.) CopyingHoststotheNetwork Configuration Panel If you want to copy one or more of the Hosts in the Hosts Database panel to the Available Hosts list in the Network Configuration panel, select the desired name(s) in the Hosts list and click on the Copy button. The selected hosts will be added to the list of Available Hosts on which you can spawn nodes Checking Network Connectivity For any compute node, you can print network connectivity information that includes the hostname, architecture, process ID, and ID of the selected compute node and all machines connected to it. The ID of the selected compute node is marked with an asterisk. The ID for the FLUENT host process is always host. The compute nodes are numbered sequentially starting from node-0. All compute nodes are completely connected. In addition, compute node 0 is connected to the host process. To obtain connectivity information for a compute node, you can use the Parallel Connectivity panel (Figure ). Parallel Show Connectivity... c Fluent Inc. November 28,

16 Parallel Processing Figure : The Parallel Connectivity Panel Indicate the compute node ID for which connectivity information is desiredinthecompute Node field, and then click on the Print button. Sample output for compute node 0 is shown below: ID Hostname O.S. PID Mach ID HW ID Name node-2 fili irix Fluent Node node-1 bofur irix Fluent Node host balin sunos Fluent Host node-0* balin sunos Fluent Node O.S is the architecture, PID is the process ID number, Mach ID is the compute node ID, and HW ID is an identifier specific to the communicator used. You can also check connectivity of a compute node in the Network Configuration panel by selecting it in the Spawned Compute Nodes list and clicking on the Connectivity button. If you click on the Connectivity button without selecting any of the Spawned Compute Nodes, theparallel Connectivity panel will open, and you can specify the node there, as described above. If you select more than one of the Spawned Compute Nodes, clicking on the Connectivity button will print connectivity information for each selected node c Fluent Inc. November 28, 2001

17 28.4 Partitioning the Grid 28.4 Partitioning the Grid Information about grid partitioning is provided in the following sections: Section : Overview of Grid Partitioning Section : Partitioning the Grid Automatically Section : Partitioning the Grid Manually Section : Grid Partitioning Methods Section : Checking the Partitions Section : Load Distribution Overview of Grid Partitioning! When you use the parallel solver in FLUENT, you need to partition or subdivide the grid into groups of cells that can be solved on separate processors (see Figure ). You can either use the automatic partitioning algorithms when reading an unpartitioned grid into the parallel solver (recommended approach, described in Section ), or perform the partitioning yourself in the serial solver or after reading a mesh into the parallel solver (as described in Section ). In either case, the available partitioning methods are those described in Section You can partition the grid before or after you set up the problem (by defining models, boundary conditions, etc.), although it is better to partition after the setup, due to some model dependencies (e.g., adaption on nonconformal interfaces, sliding-mesh and shell-conduction encapsulation). If your case file contains sliding meshes, or non-conformal interfaces on which you plan to perform adaption during the calculation, you will have to partition it in the serial solver. See Sections and for more information. Note that the relative distribution of cells among compute nodes will be maintained during grid adaption, except if non-conformal interfaces are present, so repartitioning after adaption is not required. See Section for more information. c Fluent Inc. November 28,

18 Parallel Processing If you use the serial solver to set up the problem before partitioning, the machine on which you perform this task must have enough memory to read in the grid. If your grid is too large to be read into the serial solver, you can read the unpartitioned grid directly into the parallel solver (using the memory available in all the defined hosts) and have it automatically partitioned. In this case you will set up the problem after an initial partition has been made. You will then be able to manually repartition the case if necessary. See Sections and for additional details and limitations, and Section for details about checking the partitions. Before Partitioning Domain Interface Boundary After Partitioning Partition 0 Partition 1 Figure : Partitioning the Grid c Fluent Inc. November 28, 2001

19 28.4 Partitioning the Grid Partitioning the Grid Automatically For automatic grid partitioning, you can select the bisection method and other options for creating the grid partitions before reading a case file into the parallel version of the solver. For some of the methods, you can perform pretesting to ensure that the best possible partition is performed. See Section for information about the partitioning methods available in FLUENT. Note that if your case file contains sliding meshes, or non-conformal interfaces on which you plan to perform adaption during the calculation, you will need to partition it in the serial solver, and then read it into the parallel solver, with the Case File option turned on in the Auto Partition Grid panel (the default setting). The procedure for partitioning automatically in the parallel solver is as follows: 1. (optional) Set the partitioning parameters in the Auto Partition Grid panel (Figure ). Parallel Auto Partition... Figure : The Auto Partition Grid Panel If you are reading in a mesh file or a case file for which no partition information is available, and you keep the Case File option turned on, FLUENT will partition the grid using the method displayed in the Method drop-down list. c Fluent Inc. November 28,

20 Parallel Processing If you want to specify the partitioning method and associated options yourself, the procedure is as follows: (a) Turn off the Case File option. The other options in the panel will become available. (b) Select the bisection method in the Method drop-down list. The choices are the techniques described in Section (c) You can choose to independently apply partitioning to each cell zone, or you can allow partitions to cross zone boundaries using the Across Zones check button. It is recommended that you not partition cells zones independently (by turning off the Across Zones check button) unless cells in different zones will require significantly different amounts of computation during the solution phase (e.g., if the domain contains both solid and fluid zones). (d) If you have chosen the Principal Axesor Cartesian Axes method, you can improve the partitioning by enabling the automatic testing of the different bisection directions before the actual partitioning occurs. To use pretesting, turn on the Pre-Test option. Pretesting is described in Section (e) Click on OK. If you have a case file where you have already partitioned the grid, and the number of partitions divides evenly into the number of compute nodes, you can keep the default selection of Case File in the Auto Partition Grid panel. This instructs FLUENT to use the partitions in the case file. 2. Read the case file. File Read Case... Reporting During Auto Partitioning As the grid is automatically partitioned, some information about the partitioning process will be printed in the text (console) window. If you want additional information, you can print a report from the Partition Grid panel after the partitioning is completed c Fluent Inc. November 28, 2001

21 28.4 Partitioning the Grid Parallel Partition... When you click on the Print Active Partitions or Print Stored Partitions button in the Partition Grid panel, FLUENT will print the partition ID, number of cells, faces, and interfaces, and the ratio of interfaces to faces for each active or stored partition in the console window. In addition, it will print the minimum and maximum cell, face, interface, and face-ratio variations. See Section for details. You can examine the partitions graphically by following the directions in Section Partitioning the Grid Manually Automatic partitioning in the parallel solver (described in Section ) is the recommended approach to grid partitioning, but it is also possible to partition the grid manually in either the serial solver or the parallel solver. After automatic or manual partitioning, you will be able to inspect the partitions created (see Section ) and optionally repartition the grid, if necessary. Again, you can do so within the serial or the parallel solver, using the Partition Grid panel. A partitioned grid may also be used in the serial solver without any loss in performance. Guidelines for Partitioning the Grid The following steps are recommended for partitioning a grid manually: 1. Partition the grid using the default bisection method (Principal Axes) and optimization (Smooth). 2. Examine the partition statistics, which are described in Section Your aim is to achieve small values of Interface ratio variation and Global interface ratio while maintaining a balanced load (Cell variation). If the statistics are not acceptable, try one of the other bisection methods. 3. Once you determine the best bisection method for your problem, you can turn on Pre-Test (see Section ) to improve it further, if desired. c Fluent Inc. November 28,

22 Parallel Processing 4. You can also improve the partitioning using the Merge optimization, if desired. Instructions for manual partitioning are provided below. Using the Partition Grid Panel For grid partitioning, you need to select the bisection method for creating the grid partitions, set the number of partitions, select the zones and/or registers, and choose the optimizations to be used. For some methods, you can also perform pretesting to ensure that the best possible bisection is performed. Once you have set all the parameters in the Partition Grid panel to your satisfaction, click on the Partition button to subdivide the grid into the selected number of partitions using the prescribed method and optimization(s). See above for recommended partitioning strategies. You can set the relevant inputs in the Partition Grid panel (Figure in the parallel solver, or Figure in the serial solver) in the following manner: Parallel Partition Select the bisection method in the Method drop-down list. The choices are the techniques described in Section Set the desired number of grid partitions in the Number integer number field. You can use the counter arrows to increase or decrease the value, instead of typing in the box. The number of grid partitions must be an integral multiple of the number of processors available for parallel computing. 3. You can choose to independently apply partitioning to each cell zone, or you can allow partitions to cross zone boundaries using the Across Zones check button. It is recommended that you not partition cells zones independently (by turning off the Across Zones check button) unless cells in different zones will require significantly different amounts of computation during the solution phase (e.g., if the domain contains both solid and fluid zones) c Fluent Inc. November 28, 2001

23 28.4 Partitioning the Grid Figure : The Partition Grid Panel in the Parallel Solver Figure : The Partition Grid Panel in the Serial Solver c Fluent Inc. November 28,

24 Parallel Processing 4. You can select Encapsulate Grid Interfaces if you would like the cells surrounding all non-conformal grid interfaces in your mesh to reside in a single partition at all times during the calculation. Grid interfaces must be encapsulated when the grid slides or when the mesh is adapted, so, when sliding meshes are present, the Encapsulate Grid Interfaces option will always be on. If your case file contains non-conformal interfaces on which you plan to perform adaption during the calculation, you will have to partition it in the serial solver, with the Encapsulate Grid Interfaces and Encapsulate for Adaption options turned on. 5. If you have enabled the Encapsulate Grid Interfaces option in the serial solver, the Encapsulate for Adaption option will also be available. When you select this option, additional layers of cells are encapsulated such that transfer of cells will be unnecessary during parallel adaption. 6. You can activate and control the desired optimization methods (described in Section ) using the items under Optimizations. You can activate the Merge and Smooth schemes by turning on the Do check button next to each one. For each scheme, you can also set the number of Iterations. Each optimization scheme will be applied until appropriate criteria are met, or the maximum number of iterations has been executed. If the Iterations counter is set to 0, the optimization scheme will be applied until completion, without limit on the maximum number of iterations. 7. If you have chosen the Principal Axes or Cartesian Axes method, you can improve the partitioning by enabling the automatic testing of the different bisection directions before the actual partitioning occurs. To use pretesting, turn on the Pre-Test option. Pretesting is described in Section In the Zones and/or Registers lists, select the zone(s) and/or register(s) for which you want to partition. For most cases, you will select all Zones (the default) to partition the entire domain. See below for details. 9. Click on the Partition button to partition the grid c Fluent Inc. November 28, 2001

25 28.4 Partitioning the Grid 10. If you decide that the new partitions are better than the previous ones (if the grid was already partitioned), click on the Use Stored Partitions button to make the newly stored cell partitions the active cell partitions. The active cell partition is used for the current calculation, while the stored cell partition (the last partition performed) is used when you save a case file. Partitioning Within Zones or Registers The ability to restrict partitioning to cell zones or registers gives you the flexibility to apply different partitioning strategies to subregions of a domain. For example, if your geometry consists of a cylindrical plenum connected to a rectangular duct, you may want to partition the plenum using the Cylindrical Axes method, and the duct using the Cartesian Axes method.! If the plenum and the duct are contained in two different cell zones, you can select one at a time and perform the desired partitioning, as described in Section If they are not in two different cell zones, you can create a cell register (basically a list of cells) for each region using the functions that are used to mark cells for adaption. These functions allow you to mark cells based on physical location, cell volume, gradient or isovalue of a particular variable, and other parameters. See Chapter 23 for information about marking cells for adaption. Section 23.9 provides information about manipulating different registers to create new ones. Once you have created a register, you can partition within it as described above. Note that partitioning within zones or registers is not available when the parallel version of FLUENT is used, or when Metis is selected as the partition Method. Reporting During Partitioning As the grid is partitioned, information about the partitioning process will be printed in the text (console) window. By default, the solver will print the number of partitions created, the number of bisections performed, the time required for the partitioning, and the minimum and maximum c Fluent Inc. November 28,

26 Parallel Processing! cell, face, interface, and face-ratio variations. (See Section for details.) If you increase the Verbosity to 2 from the default value of 1, the partition method used, the partition ID, number of cells, faces, and interfaces, and the ratio of interfaces to faces for each partition will also be printed in the console window. If you decrease the Verbosity to 0, only the number of partitions created and the time required for the partitioning will be reported. You can request a portion of this report to be printed again after the partitioning is completed. When you click on the Print Active Partitions or Print Stored Partitions button in the parallel solver, FLUENT will print the partition ID, number of cells, faces, and interfaces, and the ratio of interfaces to faces for each active or stored partition in the console window. In addition, it will print the minimum and maximum cell, face, interface, and face-ratio variations. In the serial solver, you will obtain the same information about the stored partition when you click on Print Partitions. See Section for details. Recall that to make the stored cell partitions the active cell partitions you must click on the Use Stored Partitions button. The active cell partition is used for the current calculation, while the stored cell partition (the last partition performed) is used when you save a case file. Resetting the Partition Parameters If you change your mind about your partition parameter settings, you can easily return to the default settings assigned by FLUENT by clicking on the Default button. When you click on the Default button, it will become the Reset button. The Reset button allows you to return to the most recently saved settings (i.e., the values that were set before you clicked on Default). After execution, the Reset button will become the Default button again Grid Partitioning Methods Partitioning the grid for parallel processing has three major goals: Create partitions with equal numbers of cells c Fluent Inc. November 28, 2001

27 28.4 Partitioning the Grid Minimize the number of partition interfaces i.e., decrease partition boundary surface area. Minimize the number of partition neighbors. Balancing the partitions (equalizing the number of cells) ensures that each processor has an equal load and that the partitions will be ready to communicate at about the same time. Since communication between partitions can be a relatively time-consuming process, minimizing the number of interfaces can reduce the time associated with this data interchange. Minimizing the number of partition neighbors reduces the chances for network and routing contentions. In addition, minimizing partition neighbors is important on machines where the cost of initiating message passing is expensive compared to the cost of sending longer messages. This is especially true for workstations connected in a network. The partitioning schemes in FLUENT use bisection algorithms to create the partitions, but unlike other schemes which require the number of partitions to be a factor of two, these schemes have no limitations on the number of partitions. For each available processor, you will create the same number of partitions (i.e., the total number of partitions will be an integral multiple of the number of processors). Bisection Methods The grid is partitioned using a bisection algorithm. The selected algorithm is applied to the parent domain, and then recursively applied to the child subdomains. For example, to divide the grid into four partitions, the solver will bisect the entire (parent) domain into two child domains, and then repeat the bisection for each of the child domains, yielding four partitions in total. To divide the grid into three partitions, the solver will bisect the parent domain to create two partitions one approximately twice as large as the other and then bisect the larger child domain again to create three partitions in total. The grid can be partitioned using one of the algorithms listed below. The most efficient choice is problem-dependent, so you can try differc Fluent Inc. November 28,

28 Parallel Processing ent methods until you find the one that is best for your problem. See Section for recommended partitioning strategies. Cartesian Axes bisects the domain based on the Cartesian coordinates of the cells (see Figure ). It bisects the parent domain and all subsequent child subdomains perpendicular to the coordinate direction with the longest extent of the active domain. It is often referred to as coordinate bisection. Cartesian Strip uses coordinate bisection but restricts all bisections to the Cartesian direction of longest extent of the parent domain (see Figure ). You can often minimize the number of partition neighbors using this approach. Cartesian X-, Y-, Z-Coordinate bisects the domain based on the selected Cartesian coordinate. It bisects the parent domain and all subsequent child subdomains perpendicular to the specified coordinate direction. (See Figure ) Cartesian R Axes bisects the domain based on the shortest radial distance from the cell centers to that Cartesian axis (x, y, orz) which produces the smallest interface size. This method is available only in 3D. Cartesian RX-, RY-, RZ-Coordinate bisects the domain based on the shortest radial distance from the cell centers to the selected Cartesian axis (x, y, or z). These methods are available only in 3D. Cylindrical Axes bisects the domain based on the cylindrical coordinates of the cells. This method is available only in 3D. Cylindrical R-, Theta-, Z-Coordinate bisects the domain based on the selected cylindrical coordinate. These methods are available only in 3D. Metis uses the METIS software package for partitioning irregular graphs, developed by Karypis and Kumar at the University of Minnesota and the Army HPC Research Center. It uses a multilevel approach c Fluent Inc. November 28, 2001

29 28.4 Partitioning the Grid! in which the vertices and edges on the fine graph are coalesced to form a coarse graph. The coarse graph is partitioned, and then uncoarsened back to the original graph. During coarsening and uncoarsening, algorithms are applied to permit high-quality partitions. Detailed information about METIS can be found in its manual [110]. Note that when using the socket version (-pnet), the METIS partitioner is not available. In this case, METIS partitioning can be obtained using the partition filter, as described below. Polar Axes bisects the domain based on the polar coordinates of the cells (see Figure ). This method is available only in 2D. Polar R-Coordinate, Polar Theta-Coordinate bisects the domain based on the selected polar coordinate (see Figure ). These methods are available only in 2D. Principal Axes bisects the domain based on a coordinate frame aligned with the principal axes of the domain (see Figure ). This reduces to Cartesian bisection when the principal axes are aligned with the Cartesian axes. The algorithm is also referred to as moment, inertial, or moment-of-inertia partitioning. This is the default bisection method in FLUENT. Principal Strip uses moment bisection but restricts all bisections to the principal axis of longest extent of the parent domain (see Figure ). You can often minimize the number of partition neighbors using this approach. Principal X-, Y-, Z-Coordinate bisects the domain based on the selected principal coordinate (see Figure ). Spherical Axes bisects the domain based on the spherical coordinates of the cells. This method is available only in 3D. Spherical Rho-, Theta-, Phi-Coordinate bisects the domain based on the selected spherical coordinate. These methods are available only in 3D. c Fluent Inc. November 28,

30 Parallel Processing 3.00e e e e e+00 Contours of Cell Partition Figure : Partitions Created with the Cartesian Axes Method 3.00e e e e e+00 Contours of Cell Partition Figure : Partitions Created with the Cartesian Strip or Cartesian X-Coordinate Method c Fluent Inc. November 28, 2001

31 28.4 Partitioning the Grid 3.00e e e e e+00 Contours of Cell Partition Figure : Partitions Created with the Principal Axes Method 3.00e e e e e+00 Contours of Cell Partition Figure : Partitions Created with the Principal Strip or Principal X-Coordinate Method c Fluent Inc. November 28,

32 Parallel Processing 3.00e e e e e+00 Contours of Cell Partition Figure : Partitions Created with the Polar Axes or Polar Theta- Coordinate Method Optimizations Additional optimizations can be applied to improve the quality of the grid partitions. The heuristic of bisecting perpendicular to the direction of longest domain extent is not always the best choice for creating the smallest interface boundary. A pre-testing operation (see Section ) can be applied to automatically choose the best direction before partitioning. In addition, the following iterative optimization schemes exist: Smooth attempts to minimize the number of partition interfaces by swapping cells between partitions. The scheme traverses the partition boundary and gives cells to the neighboring partition if the interface boundary surface area is decreased. (See Figure ) Merge attempts to eliminate orphan clusters from each partition. An orphan cluster is a group of cells with the common feature that each cell within the group has at least one face which coincides with c Fluent Inc. November 28, 2001

33 28.4 Partitioning the Grid an interface boundary. (See Figure ) Orphan clusters can degrade multigrid performance and lead to large communication costs. Figure : The Smooth Optimization Scheme Figure : The Merge Optimization Scheme In general, the Smooth and Merge schemes are relatively inexpensive optimization tools. Pretesting If you choose the Principal Axes or Cartesian Axes method, you can improve the bisection by testing different directions before performing the actual bisection. If you choose not to use pretesting (the default), FLU- ENT will perform the bisection perpendicular to the direction of longest domain extent. c Fluent Inc. November 28,

34 Parallel Processing If pretesting is enabled, it will occur automatically when you click on the Partition buttoninthepartition Grid panel, or when you read in the grid if you are using automatic partitioning. The bisection algorithm will test all coordinate directions and choose the one which yields the fewest partition interfaces for the final bisection. Note that using pretesting will increase the time required for partitioning. For 2D problems partitioning will take 3 times as long as without pretesting, and for 3D problems it will take 4 times as long. Using the Partition Filter As noted above, you can use the METIS partitioning method through a filter in addition to within the Auto Partition Grid and Partition Grid panels. To perform METIS partitioning on an unpartitioned grid, use the File/Import/Partition/Metis... menu item. File Import Partition Metis...! FLUENT will use the METIS partitioner to partition the grid, and then read the partitioned grid into the solver. The number of partitions will be equal to the number of processes. You can then proceed with the model definition and solution. Direct import to the parallel solver through the partition filter requires that the host machine has enough memory to run the filter for the specified grid. If not, you will need to run the filter on a machine that does have enough memory. You can either start the parallel solver on the machine with enough memory and repeat the process described above, or run the filter manually on the new machine and then read the partitioned grid into the parallel solver on the host machine. To manually partition a grid using the partition filter, enter the following command: utility partition input-filename partition-count output-filename where input-filename is the filename for the grid to be partitioned, partition-count is the number of partitions desired, and output-filename c Fluent Inc. November 28, 2001

35 28.4 Partitioning the Grid is the filename for the partitioned grid. You can then read the partitioned grid into the solver (using the standard File/Read/Case... menu item) and proceed with the model definition and solution. When the File/Import/Partition/Metis... menu item is used to import an unpartitioned grid into the parallel solver, the METIS partitioner partitions the entire grid. You may also partition each cell zone individually, using the File/Import/Partition/Metis Zone... menu item. File Import Partition Metis Zone... This method can be useful for balancing the work load. For example, if a case has a fluid zone and a solid zone, the computation in the fluid zone is more expensive than in the solid zone, so partitioning each zone individually will result in a more balanced work load Checking the Partitions After partitioning a grid, you should check the partition information and examine the partitions graphically. Interpreting Partition Statistics You can request a report to be printed after partitioning (either automatic or manual) is completed. In the parallel solver, click on the Print Active Partitions or Print Stored Partitions button in the Partition Grid panel. In the serial solver, click on the Print Partitions button. FLUENT distinguishes between two cell partition schemes within a parallel problem: the active cell partition and the stored cell partition. Initially, both are set to the cell partition that was established upon reading the case file. If you re-partition the grid using the Partition Grid panel, the new partition will be referred to as the stored cell partition. To make it the active cell partition, you need to click on the Use Stored Partitions button in the Partition Grid panel. The active cell partition is used for the current calculation, while the stored cell partition (the last partition performed) is used when you save a case file. This distinction is made mainly to allow you to partition a case on one machine or network of machines and solve it on a different one. Thanks to the two separate c Fluent Inc. November 28,

36 Parallel Processing partitioning schemes, you could use the parallel solver with a certain number of compute nodes to subdivide a grid into an arbitrary different number of partitions, suitable for a different parallel machine, save the case file, and then load it into the designated machine. When you click Print Partitions in the serial solver, you will obtain information about the stored partition. The output generated by the partitioning process includes information about the recursive subdivision and iterative optimization processes. This is followed by information about the final partitioned grid, including the partition ID, number of cells, number of faces, number of interface faces, ratio of interface faces to faces for each partition, number of neighboring partitions, and cell, face, interface, neighbor, mean cell, face ratio, and global face ratio variations. Global face ratio variations are the minimum and maximum values of the respective quantities in the present partitions. For example, in the sample output below, partitions 0 and 3 have the minimum number of interface faces (10), and partitions 1 and 2 have the maximum number of interface faces (19); hence the variation is Your aim is to achieve small values of Interface ratio variation and Global interface ratio while maintaining a balanced load (Cell variation) c Fluent Inc. November 28, 2001

37 28.4 Partitioning the Grid >> Partitions: P Cells I-Cells Cell Ratio Faces I-Faces Face Ratio Neighbors Partition count = 4 Cell variation = ( ) Mean cell variation = ( -1.1% - 1.1%) Intercell variation = (10-19) Intercell ratio variation = ( 7.3% %) Global intercell ratio = 10.7% Face variation = ( ) Interface variation = (10-19) Interface ratio variation = ( 4.5% - 8.7%) Global interface ratio = 3.4% Neighbor variation = (1-2) Computing connected regions; type ^C to interrupt. Connected region count = 4 Note that partition IDs correspond directly to compute node IDs when a case file is read into the parallel solver. When the number of partitions in a case file is larger than the number of compute nodes, but is evenly divisible by the number of compute nodes, then the distribution is such that partitions with IDs 0 to (M 1) are mapped onto compute node 0, partitions with IDs M to (2M 1) onto compute node 1, etc., where M is equal to the ratio of the number of partitions to the number of compute nodes. Examining Partitions Graphically To further aid interpretation of the partition information, you can draw contours of the grid partitions, as illustrated in Figures Display Contours... To display the active cell partition or the stored cell partition (which are described above), select Active Cell Partition or Stored Cell Partition c Fluent Inc. November 28,

38 Parallel Processing! in the Cell Info... category of the Contours Of drop-down list, and turn off the display of Node Values. (See Section for information about displaying contours.) If you have not already done so in the setup of your problem, you will need to perform a solution initialization in order to use the Contours panel Load Distribution If the speeds of the processors that will be used for a parallel calculation differ significantly, you can specify a load distribution for partitioning, using the load-distribution text command. parallel partition set load-distribution For example, if you will be solving on three compute nodes, and one machine is twice as fast as the other two, then you may want to assign twice as many cells to the first machine as to the others (i.e., a load vector of (2 1 1)). During subsequent grid partitioning, partition 0 will end up with twice as many cells as partitions 1 and 2.! Note that for this example, you would then need to start up FLUENT such that compute node 0 is the fast machine, since partition 0, with twice as many cells as the others, will be mapped onto compute node 0. Alternatively, in this situation, you could enable the load balancing feature (described in Section ) to have FLUENT automatically attempt to discern any difference in load among the compute nodes. If you adapt a grid that contains non-conformal interfaces, and you want to rebalance the load on the compute nodes, you will have to save your case and data files after adaption, read the case and data files into the serial solver, repartition using the Encapsulate Grid Interfaces and Encapsulate for Adaption options in the Partition Grid panel, and save case and data files again. You will then be able to read the manually repartitioned case and data files into the parallel solver, and continue the solution from where you left it c Fluent Inc. November 28, 2001

39 28.5 Checking and Improving Parallel Performance 28.5 Checking and Improving Parallel Performance To determine how well the parallel solver is working, you can measure computation and communication times, and the overall parallel efficiency, using the performance meter. You can also control the amount of communication between compute nodes in order to optimize the parallel solver, and take advantage of the automatic load balancing feature of FLUENT Checking Parallel Performance The performance meter allows you to report the wall clock time elapsed during a computation, as well as message-passing statistics. Since the performance meter is always activated, you can access the statistics by printing them after the computation is completed. To view the current statistics, use the Parallel/Timer/Usage menu item. Parallel Timer Usage Performance statistics will be printed in the text window (console). To clear the performance meter so that you can eliminate past statistics from the future report, use the Parallel/Timer/Reset menu item. Parallel Timer Reset Optimizing the Parallel Solver Increasing the Report Interval In FLUENT, you can reduce communication and improve parallel performance by increasing the report interval for residual printing/plotting or other solution monitoring reports. You can modify the value for Reporting Interval in the Iterate panel.! Solve Iterate... Note that you will be unable to interrupt iterations until the end of each report interval. c Fluent Inc. November 28,

40 Parallel Processing Load Balancing A dynamic load balancing capability is available in FLUENT. The principal reason for using parallel processing is to reduce the turnaround time of your simulation, ideally by a factor proportional to the collective speed of the computing resources used. If, for example, you were using four CPUs to solve your problem, then you would expect to reduce the turnaround time by a factor of four. This is of course the ideal situation, and assumes that there is very little communication needed among the CPUs, that the CPUs are all of equal speed, and that the CPUs are dedicated to your job. In practice, this is often not the case. For example, CPU speeds can vary if you are solving in parallel on a heterogeneous collection of workstations, other jobs may be competing for use of one or more of the CPUs, and network traffic either from within the parallel solver or generated from external sources may delay some of the necessary communication among the CPUs.! If you enable dynamic load balancing in FLUENT, the load across the computational and networking resources will be monitored periodically. If the load balancer determines that performance can be improved by redistributing the cells among the compute nodes, it will automatically do so. There is a time penalty associated with load balancing itself, and so it is disabled by default. If you will be using a dedicated homogeneous resource, or if you are using a heterogeneous resource but have accounted for differences in CPU speeds during partitioning by specifying a load distribution (see Section ), then you may not need to use load balancing. Note that when non-conformal interfaces are present, or the shell conduction model is used, you will not be able to turn on load balancing. To enable and control FLUENT s automatic load balancing feature, use the Load Balance panel (Figure ). Load balancing will automatically detect and analyze parallel performance, and redistribute cells between the existing compute nodes to optimize it. Parallel Load Balance... The procedure for using load balancing is as follows: c Fluent Inc. November 28, 2001

41 28.5 Checking and Improving Parallel Performance Figure : The Load Balance Panel 1. Turn on the Load Balancing option. 2. Select the bisection method to create new grid partitions in the Partition Method drop-down list. The choices are the techniques described in Section As part of the automatic load balancing procedure, the grid will be repartitioned into several small partitions using the specified method. The resulting partitions will then be distributed among the compute nodes to achieve a more balanced load. 3. Specify the desired Balance Interval. When a value of 0 is specified, FLUENT will internally determine the best value to use, initially using an interval of 25 iterations. You can override this behavior by specifying a non-zero value. FLUENT will then attempt to perform load balancing after every N iterations, where N is the specified Balance Interval. You should be careful to select an interval that is large enough to outweigh the cost of performing the load balancing operations. Note that you can interrupt the calculation at any time, turn the load balancing feature off (or on), and then continue the calculation. c Fluent Inc. November 28,