Cornell Theory Center 1

Transcription

1 Cornell Theory Center Cornell Theory Center (CTC) is a high-performance computing and interdisciplinary research center at Cornell University. Scientific and engineering research projects supported by CTC represent a variety of disciplines, including biology, behavioral and social sciences, computer science, engineering, geosciences, mathematics, physical sciences, and business. CTC is unique among HPC centers in that it is a showcase for Windows-based high-performance computing. In 1999, we successfully moved our research community from a traditional, proprietary supercomputer running UNIX to an industry-standard cluster complex. Microsoft and Dell are major supporters of CTC, which is directed by computer scientist Thomas F. Coleman. 3/12/03 Cornell Theory Center 1 Case Studies Plant breeders and molecular biologists generating detailed information on the genetic makeup of crops e.g. rice and wheat Management model for manufacturing that helps managers find best balance between capacity, inventory, and subcontracting New stochastic path approach for modeling protein dynamics in more meaningful time scales New social science research made possible by developing new statistical methods for analyzing census data Simulation of crack growth and strength evaluation of airplane components 3/12/03 Cornell Theory Center 2 Compute Power Over 700 Processors Giganet Switch (100 GB/sec) interconnect 100 Mb switched Ethernet 1-4 GB Ram/Node, GB Disk/Node Serial, Dual, and Quad processors Dell PowerEdge Servers Pentium processors, 600 Mhz 2.4Ghz 256 KB - 2 MB Cache/Processor (SMP) Tivoli Backup System Microsoft Distributed File System Fileservers, 5 Terabytes Storage 11/ processor cluster in top gigaflops over Ghz Linpack benchmark 3/12/03 Cornell Theory Center 3 Cornell Theory Center 1

2 Visualization Consultation on how to visualize data Creation of animations, videos, publication graphics Independent study opportunities for undergraduate students Training in the use of OpenDX Video production suite Three-wall Windows CAVE: Virtual reality environment (show VRML files) Write your own with OpenGL or OpenInventor Free by appointment with Computer Lab of Visualization Workstations /12/03 Cornell Theory Center 4 Software See online list for software available on the login nodes, various types of compute nodes and the Visualization Collaboratory. This list is updated as new packages are added. Software is added by researcher's request if feasible. C/C++ compilers: Microsoft Visual C++, Intel C/C++, Gnu gcc/g++ Fortran compilers: Intel Fortran, PGF77, Compaq Visual Fortran Libraries: MPI/PRO parallel programming, Numerical Libraries Applications software: Computational chemistry, Data analysis, Finite element modeling, Fluid dynamics, Visualization tools, /12/03 Cornell Theory Center 5 Allocations Research allocation: Submit proposals at any time. Undergraduate or graduate students must have their requests submitted by a faculty advisor. Non-renewable six-month Exploratory: Allocations of 100 hours. Simplified application. Charging is calculated using wall-clock hours; i.e. you will be charged for the time nodes are reserved for your use, not for the time you are actually using them. Wall-clock hours on all of the node types are charged the same flat wall-clock rate, regardless of number of processors. Parallelize! Charges for compute nodes only, not for login nodes or the collaboratory machines /12/03 Cornell Theory Center 6 Cornell Theory Center 2

3 Access Connect to a login node (ctclogin1, ctclogin2, ) to compile and submit jobs. The login nodes are SSH servers, download an SSH client. Login.bat is run automatically. For full windows desktop functionality from your machine, use a terminal server, Microsoft Terminal Server Client for Windows, Citrix ICA for non-windows. For command line access, use telnet: ssh or telnet ctclogin1.tc.cornell.edu 3/12/03 Cornell Theory Center 7 Development Environment Programs can be compiled on your Windows machine, in the Collaboratory in Rhodes Hall, or from one of the login nodes reached via Terminal Server or telnet. Work from the command line or in a GUI environment. GUI Development environment: Start Programs Microsoft Visual Studio.NET Microsoft Visual Studio.NET Command line: Start Run cmd (bring up a command line window) setup_intelc32 (set environment variables using setup script **) cl hello.c (compile serial) mpicc hello.c (compile MPI) ** To see the full list of setup scripts, issue: call "H:\CTC Tools\setup_list.bat 3/12/03 Cornell Theory Center 8 Compute Environment Batch Scheduler Login ctclogin1 ctclogin2 CCS Compute ctclogin H: Fileserver 3/12/03 Cornell Theory Center 9 Cornell Theory Center 3

4 Serial Batch Job Script 1. Prepare your executable 2. Write a batch script file 3. From one of the login nodes, submit the script to the scheduler 4. Check the status of the queue, your job, etc The remainder of the steps can be done by hand or script: 5. When your job has started: copy files to the local drive run the program 6. When your job has ended: Copy any output files back to H: clean up the drive end the job Login ctclogin1 ctclogin2 CCS Compute ctclogin H: 3/12/03 Cornell Theory Center 10 Parallel Batch Job Script The main batch script runs on the Master node Start the script with the four required REM CCS keyword commands Create the machines file on of the master node Use mpirun to run setup.bat on all nodes listed in the machines file Login Node Create a working subfolder on the compute node drive Copy the executable (and data files, if necessary) to each node Copy any files needed only on the Master node Move the machines file from to \your_userid H: Use mpirun to run the MPI program executable on all nodes listed in the machines file Your_program.exe runs on all the nodes in the machines file Use mpirun to run cleanup.bat on all nodes listed in the machines file Copy the output files from each node to H: Cleanup the working subdirectory on each node Release the nodes Master Node Worker Node Worker Node Worker Node 3/12/03 Cornell Theory Center 11 Parallel Job on a Single Node (page 1 of 2) REM CCS account = your_userid REM CCS type = batch REM CCS nodes = 1 REM CCS minutes = 10 REM CCS requirements = 1@development mkdir %USERNAME% cd %USERNAME% del /Q /S \%USERNAME%\*.* Call machinemaker (in CTC Tools, therefore already in your path) to produce a machines list, listing the node(s) your job has been allocated. Note: if you have more than one job running, you can assure that you'll create the machines file for the right jobid by issuing machinemaker jobid call machinemaker 3/12/03 Cornell Theory Center 12 Cornell Theory Center 4

5 Parallel Job on a Single Node (page 2 of 2) REM Copy the executable and data files to the node copy H:\users\%USERNAME%\lab\wavesend.exe \%USERNAME% copy H:\users\%USERNAME%\lab\wave.in \%USERNAME% Run the MPI program with mpirun. Set -np to the number of tasks. Set -wd to the working directory in the case where your files are not in your current directory. mpirun -wd \%USERNAME% -np 2 wavesend.exe 1>wave.out 2>wave.err REM Copy any output created by the master task back to the H drive. copy \%USERNAME%\wave.* H:\users\%USERNAME%\lab REM Cleanup the working subdirectory on each node del /Q /S \%USERNAME%\*.* REM Release the nodes ccrelease 3/12/03 Cornell Theory Center 13 Parallel Job on Multiple (page 1 of 4) REM CCS account = your_userid REM CCS type = batch REM CCS nodes = 4 REM CCS minutes = 10 REM CCS requirements = 4@development REM Move to the T drive call machinemaker Run machinemaker to create a file called "machines", which contains a list of machines allocated to this job REM Set the communication mode to TCP or VIA set MPI_COMM=VIA 3/12/03 Cornell Theory Center 14 Parallel Job on Multiple (page 2 of 4) Use mpirun to call a batch script called setup.bat. Setup.bat to copy the executable, data files, etc onto the nodes listed in the machines file. When calling setup.bat, set -np to the number of nodes, not the number of tasks. mpirun -np 4 H:\users\%USERNAME%\lab\ex3setup.bat REM setup.bat REM Create a working subdirectory on each node mkdir your_userid cd your_userid del /Q /S \your_userid\*.* REM Copy the executable (and data files, if necessary) to each node copy H:\users\your_userid\lab\wavesend.exe \your_userid 3/12/03 Cornell Theory Center 15 Cornell Theory Center 5

6 Parallel Job on Multiple (page 3 of 4) REM The input file is only needed on the master node copy H:\users\%USERNAME%\lab\wave.in \%USERNAME% Move the machines file from to \your_userid. It couldn't be put there earlier. cd %USERNAME% move \machines \%USERNAME% Run the MPI program with mpirun. Set -np to the number of tasks. Set -wd to the working directory used by mpirun. mpirun -wd \%USERNAME%\ -np 8 wavesend.exe 1>wave.out 2>wave.err Copy any output created by the master task back to H:. copy \%USERNAME%\prog.* H:\users\%USERNAME%\lab 3/12/03 Cornell Theory Center 16 Parallel Job on Multiple (page 4 of 4) Next another script (cleanup.bat) is run all nodes to both copy any other output files back to the H drive and to clean up the subdirectory. When calling cleanup.bat, set -np to the number of nodes,not the number of tasks. mpirun -np 4 H:\users\%USERNAME%\lab\ex3cleanup.bat REM Release the nodes ccrelease REM In this example, only the master node has output files. copy \your_userid\wave.* H:\users\your_userid\lab REM Cleanup the working subdirectory on each node del /Q /S \your_userid\*.* 3/12/03 Cornell Theory Center 17 MCP Fileserver (H:) Standard copy: Fileserver is accessed N times Copy with mcp Fileserver is accessed 1 time 3/12/03 Cornell Theory Center 18 Cornell Theory Center 6

7 More Information: Consulting - Send questions about use of the Velocity Cluster to consult@tc.cornell.edu or call and ask for a consultant. Can answer specific questions, help with problems, discuss strategies and code planning. Strategic users (faculty with proven application, suitable for scaling) have access to more in-depth long-term consulting help /12/03 Cornell Theory Center 19 Cornell Theory Center 7