Running Parallel Simulations and Enabling Science Gateways with the NSF MATLAB Experimental Computing Resource at Cornell

Transcription

1 Running Parallel Simulations and Enabling Science Gateways with the NSF MATLAB Experimental Computing Resource at Cornell Steve Lantz, Cornell Center for Advanced Computing Susan Mehringer, Cornell Center for Advanced Computing Presented at SC10, New Orleans, LA November, 2010 The MATLAB on the TeraGrid experimental computing resource is funded by NSF grant in partnership with Purdue University, Dell, The MathWorks, and Microsoft.

2 MATLAB on the TeraGrid This is an effort to provide a large parallel MATLAB resource to a national (and international) community in a secure, useable manner. Several different hardware and software components make up the system. These integrate with the MATLAB client at different levels. All functions are provided by various services, meaning you never actually log on to any CAC systems. The client software simply makes requests to CAC systems. 2

3 High-Level Process Security is managed via short-lived certificates. When you log in to the system, you trade a username/password for a certificate that allows you to use the services. File transfer service enables you to move files through a specialized FTP server to a network file system that is mounted on all compute nodes. Job submission service enables you to submit and query jobs on the cluster; these jobs are executed by MATLAB workers on the compute nodes. 3

4 Hardware View 1. Retrieve certificate 2. Upload files to storage via GridFTP 3. Submit job to run MATLAB workers on cluster 4. Download files via GridFTP CAC 10GB Interconnect HPC 2008 Head Node MyProxy Server GridFTP Server DataDirect Networks 9700 Storage Windows Server

5 Software View File movement and job submission interactions are largely hidden by software integrated with MATLAB. CAC s client code for MATLAB is a mix of Java and.m files that enable access to the TUC cluster directly from your MATLAB client through the PCT generic scheduler interface. Client code will communicate as needed with server-side software to run your parallel jobs on TUC, the 512-core cluster devoted to parallel MATLAB applications. 5

6 parfor createjob submit matlabpool getalloutputarguments Apache CXF Certificate JSDL SSL Management MyProxy GridFTP JGlobus CoG 6

7 A Word on Security Logging on to MyProxy returns a short-lived X.509 certificate that is used to authenticate to services. This allows any TeraGrid user to access the system using their username/password on a TeraGrid MyProxy server. Most users will use the CAC MyProxy server. Job submission and status information is accessed via a web service call that is secured by a client-certificate SSL (or TLS) connection. Your data and job requests are transferred over secure channels. 7

8 GridFTP GridFTP is an extension of the standard File Transfer Protocol, developed as part of the Globus Toolkit. GridFTP provides two key extensions that the CAC client code uses: GSI Security The Grid Security Infrastructure provides file transfer authentication and encryption and interoperates with MyProxy X.509 certificates. Parallel Transfers (extended block mode) Makes use of multiple simultaneous connections so a higher percentage of available bandwidth can be used. 8

9 A Note on the Platform The compute nodes that run the MATLAB jobs are running Windows HPC 2008 (64 bit). Since a minority of people are running a Win64 platform, any files requiring compilation (e.g., mex files) will likely need to be recompiled on TUC. MATLAB is relatively resilient to paths with the wrong direction of slashes, but the difference can cause problems. C:\Users\naw47\myfiles\this.dat /home/naw47/myfiles/this.dat Windows path Mac, Linux path 9

10 Support As a funded project, the system is free to use for research applications. We will ask for information on your project so that we can learn who we are supporting and how to best address problems. We also provide consulting support for the system. Troubleshooting Guidance on optimizing your application General help with parallel MATLAB 10

11 Installing the CAC Parallel MATLAB Client Code

12 Installation Overview 1. Check that prerequisites are met 2. Download the CAC client code 3. Modify your MATLAB classpath.txt file 4. Modify one function in the CAC client code 5. Register your Certificate 6. Set paths, runtime, etc 7. Run test jobs 12

13 Prerequisites Linux, Mac, or Windows operating system on the client machine MATLAB Release 2009a or 2009b or 2010a MATLAB Parallel Computing Toolbox Obtained access via submitting the Interest Form found at

14 Terminology MATLABROOT: The MATLAB installation directory >> matlabroot Common locations are: Vista/XP/7: C:\Program Files\MATLAB\R2009a Mac: /Applications/MATLAB_R2009a.app Linux: /opt/matlab/r2009a or /usr/local/matlab/r2009a CACHOME: Wherever you install the CAC client code Be sure to substitute your folder path for CACHOME in all installation steps. Can be named something else. 14

15 Helpful Links General: CAC client code, download and installation FAQ: Help

16 Download CAC Client Code Choose a name and location for CACHOME. You will need write permissions. Some good choices: Windows: c:\username\cac Mac: /Users/username/cac Linux: /home/username/cac Download and extract the.zip file: Unpack it into CACHOME. You should end up with a folder which contains.m files and subdirectories. 16

17 classpath.txt: Java Libraries The CAC client code is heavily dependant on a series of Java libraries for functionality. The CACHOME/lib/*.jar files must be added to MATLAB s java classpath, in the text file classpath.txt. Find the location of classpath.txt: >> which classpath.txt 17

18 classpath.txt: Modifications maci64=$matlabroot/java/jarext/aquadecorations.jar After this line add 12 lines CACHOME\lib\littlejohn.jar Note these are Windows slashes, reverse them for Mac and Unix. Replace CACHOME with your install path, and it must be an absolute path. CACHOME\lib\bcprov-jdk jar CACHOME\lib\bcprov-jdk jar CACHOME\lib\cog-jglobus jar CACHOME\lib\commons-logging jar CACHOME\lib\cryptix-asn1.jar CACHOME\lib\cryptix.jar CACHOME\lib\cryptix32.jar CACHOME\lib\cxf jar CACHOME\lib\log4j jar CACHOME\lib\not-yet-commons-ss jar CACHOME\lib\puretls.jar Comment out one line: ## $matlabroot/java/jarext/ice/ib6https.jar 18

19 classpath.txt: Testing 1. Restart MATLAB, then: 2. List the paths of all of the jar files that MATLAB knows about. Do you see the 12 lines you added? >> javaclasspath 3. Are you using the classpath.txt file you expected? >> which classpath.txt 4. Test that classpath.txt is set up properly. >> addpath('littlejohnhome/contrib'); >> updatecontrib(); >> caccheckclasspath(); 19

20 classpath.txt: But, what if You are not administrator on the machine? You must run explicitly as the administrator when editing classpath.txt for the changed to be saved, since it affects the global MATLAB install. If this is not feasible, e.g. you are on a multi-user system: Identify your startup directory: >> userpath Place a classpath.txt file in your startup directory. This file is userspecific and will only affect your MATLAB environment. MATLAB looks first in the startup directory for a classpath.txt file, then the default directory, using whichever it finds first. nv/f html 20

21 cacsched.m: Modifications Start MATLAB Edit CACHOME\cacsched.m. Change USERNAME to your CAC username in line 24. Add the CAC client code to your MATLAB path: >> addpath('cachome'); Run cacsched to set up the scheduler object, sched. This object is passed to the createjob functions in order to initiate jobs. The scheduler settings will be output. >> cacsched 21

22 cacsched.m: Optional Review the following line from cacsched.m: set(sched,'datalocation',fullfile(ljhome, 'jobs')); The communication between the client MATLAB and the scheduler is file based. This means each job submission creates a large number of files which need to be stored somewhere on the client machine. The default is to set it to CACHOME\jobs. For a different location, change the line to an explicit path, e.g. set(sched,'datalocation','/home/myuserid/myjobslocation'); 22

23 Register your Certificate Download the CAC server certificates and register your certificate with us. >> cacregistercertificate(); Follow the dialogue box instructions Run this again any time you change your password It can take up to two minutes to complete 23

24 Job Submission Settings ClusterInfo.setQueueName('Quick'); only use this for 10 minutes and 16 cores or less These settings will be in effect for all subsequent job submissions, until you change them: >> ClusterInfo.setWallTime(10); set your wall time limit to 10 minutes >> ClusterInfo.getWallTime show your current wall time setting >> help ClusterInfo see more examples 24

25 Installation Testing runtests.m, found in CACHOME, is a tool which performs a series of functionality tests: >> runtests(sched,1); % run local tests on file and path settings >> runtests(sched,2); % run test on the file transfer system and on scheduler communication. >> runtests(sched,3); % run sample MATLAB jobs to the cluster to ensure that both parallel and distributed tests are functioning. >> runtests(sched,0); % run all tests 25

26 >> addpath('cachome'); >> cacsched Future Sessions >> addpath('cachome/contrib'); >> updatecontrib(); ClusterInfo.setQueueName('Quick'); % Or rely on setting from previous session >> ClusterInfo.setWallTime(10); % Or rely on setting from previous session >> submityourjob(sched); % submityourjob is your job 26

27 How to Submit a Job

28 Next Steps At this point, you have a fully operational install of the CAC client code for parallel MATLAB. Your next step should be to take a look in the examples directory to start seeing how to take advantage of TUC. cacsubmit super simple distributed job example cacparsubmit simple example of a parallel (MPI) job pooljobremote MATLAB pool example cacnonblocksubmit example of submitting a nonblocking job (avoiding waitforstate) 28

29 Using the PCT MATLAB s Parallel Computing Toolbox is the client-side toolbox that enables parallelism (including using TUC). PCT provides a set of interfaces that allow us (CAC) to write implementations of PCT functions that talk to TUC but look the same to you (the user) as when run locally. Parallel resources are selected either by using a named configuration or by using the findresource function. Either way, PCT function calls must be tied to specific implementations to provide resource-specific functionality. You don t ever call the underlying functions directly. 29

30 findresource In our examples (and in general practice) we call the findresource function via a script called cacsched.m. If you examine cacsched, you ll see we also tie the PCT interface functions to specific functions provided by CAC. 30

31 Using a Configuration 31

32 Jobs and Tasks findresource creates a scheduler object, which allows you to create Jobs. In PCT, Jobs are containers for Tasks, which are where the actual work is. j=createjob(sched) sched Scheduler Object j=createparalleljob(sched) Jobs(24) Jobs(25) createtask(j, ) Tasks(1) somefunction(x) createtask(j, ) Tasks(2) otherfunction(y) createtask(j, ) Tasks(1) myfunction(z) 32

33 Distributed Jobs There are three types of jobs in the PCT: distributed, parallel, and pool. Distributed jobs have one or more one-core tasks and no communication between tasks. Thus, each task could be run as a one-core job through a batch scheduler. These are useful for EP work or for shifting lengthy jobs to TUC. 33

34 Parallel and Pool Jobs Parallel and Pool jobs are multi-core; communication between cores is possible. These jobs have just one task! The number of cores must be given. The task function is responsible for implementing the actual parallelism using MPI_Rank logic (or parfor/spmd/labindex for pool jobs). 34

35 State of Jobs After a job is submitted, job.state is just one of several different ways to learn the state of the job. waitforstate is a PCT interface to block on job state; it s problematic for long running jobs or jobs that fail. cacnonblockinggetjobstates is an optional, non-pct interface that offers more control. 35

36 Retrieving Results Once your job completes, you need to get the results in two steps: (1) download files, (2) load into workspace. getalloutputarguments returns cell array a{task,output} a{1,2} = Task 1, second output 36

37 Helpers The CAC client code provides a number of functions beyond the PCT interface which should be helpful to you. The hands-on labs will take advantage of these functions. gridftp() creates an object whose methods are, in effect, a command-line interface to the TUC file storage. littlejohnlog/qpeek monitor the status of a running job. littlejohnlog is a server-side function that writes data to a file that qpeek reads. geterrors/getoutput pretty-print any errors your tasks had, as well as the command-line output. 37

38 Putting it All Together We can control which resource is used to execute the job simply by swapping out the scheduler object! 38

39 Parallelizing a Pool Code As we have seen, converting code to run remotely as a distributed job is fairly trivial. All you really need is to do createtask on your function (maybe in a loop). Pool jobs are not hard, either. Let s take a code that opens a pool on a multi-core workstation and alter it to exploit the many cores on TUC. The basic process: 1) Modify the pool function to run on TUC 2) Write the submitter or driver script 3) Script the movement of any needed files 39

40 Pool Code Our example code opens a matlabpool, reads an input file, then uses a parfor loop to execute the peakpicking algorithm in parallel. 40

41 Modifications to the Pool Code More outputs are needed Pool commands are removed Absolute paths are best for I/O Graphics may be moved off to the client, or may be dumped to a file 41

42 Submitter Script The submitter or driver script sets up the pool job It starts up the matlabpool automatically (8 labs here) 42

43 Moving the Files Both the task function and the datafile must be present on the remote server. We ll use gridftp to take care of it. The submitter also sets PathDependencies for the job. 43

44 Parallel Jobs PCT supports basic MPI commands inside parallel jobs. Initialization is done for you (no MPI_Init) Size and rank are available from the start of the job; numlabs = MPI_Comm_size, labindex = MPI_Comm_rank 44

45 More on Parallel Jobs All the basic communication methods are available: Send, Receive, Broadcast, Barrier, gop (gather) Source and tag are the same as in MPI, but MATLAB figures out data formats for you. labsend(data,destination,[tag]); labreceive(source,tag); labreceive(); %take anything Co-distributed arrays are sliced across the workers so that huge matrices can be operated on. 45

46 File Transfer

47 Hardware View 1. Retrieve certificate 2. Upload files to storage via GridFTP 3. Submit job to run MATLAB workers on cluster 4. Download files via GridFTP CAC 10GB Interconnect HPC 2008 Head Node MyProxy Server GridFTP Server DataDirect Networks 9700 Storage Windows Server

48 File Transfer The basic job submit operation specifies that the program will run on a remote server. When it runs, the functions and data must be available. j = createjob(sched); createtask(j,@rand,1,{3,3}); submit(j); waitforstate(j); a = getalloutputarguments(j); This example works as-is because rand is a built-in MATLAB function. It is always on the MATLAB path. 48

49 File Transfer Have a custom function and/or require a datafile? j = createjob(sched); createtask(j,@myfunction,1,{3,3}); submit(j); waitforstate(j); a = getalloutputarguments(j); myfunction.m does not exist on the remote computer. Transfer this file and get it added to the path. 49

50 FileDependencies The MATLAB FileDependencies property will move the files for you Best for smaller projects with only a couple of files Specify directories and files the worker will need. All files and directory structure will be copied; file transfer occurs for each worker running a task for that particular job on a machine set(j,'filedependencies',{'/home/username/src/myfuncti on.m', '/home/username/data/dfile.mat'); 50

51 Move the Files Yourself FileDependencies is best for smaller projects with only a couple of files Alternative: 1.Move the file(s) 2.Add the path to the worker sessions 51

52 1. Move the Files First move the file(s) needed by the job: sendfiletocac('filename.m'); sendfiletocac('filename') super simple method for dumping a single file into your home directory on TUC. senddirtocac('mydir','tucdir') Recursively move a directory and its contents to TUC. 52

53 2. Add the path On your laptop/workstation, you commonly issue addpath('path/to/file') statements. The same is true when running MATLAB on TUC, but: The task function must be on the startup path of MATLAB. You may enter addpath and cd statements into your task function, but first your function must be available. We will use PathDependencies to make our task function available. 53

54 PathDependencies PathDependencies is a property of the Job object that allows you to issue addpath statements on TUC before calling your task. Assuming the file has been moved to \\storage01\matlab\username\myprojectdir Specify the path dependency in your job submission script: set(j,'pathdependencies',{'\\storage01\matlab\username MyProjectDir'}); 54

55 PathDependencies 55

56 Scripted Solution Both send*tocac methods are primarily for one time use, best for moving a big directory of data files, testing, or copying a single file. The gridftp interface is more flexible. It allows you to interactively move files to TUC as well as write scripts that move files. For projects that involve more than one or two source files, we recommend writing a prep function which ensures that the most up-to-date functions are available on TUC. 56

57 Then add PathDependencies to the job in the submission script: 57

58 Lab Source files: calclatlongdistance.m degrees2radians.m Data file: Airports_boardings.txt Task function: addpath_remote.m Batch script: addpath_submit.m Using this set of files we will work with FileDependencies PathDependencies GridFTP 58

59 Debugging

60 Debugging Debugging a remote process is always difficult. The situation on TUC is no different. Errors must be caught, captured and returned to the client machine to resolve. MATLAB generally captures any errors thrown by a task and stores them as a MException in the task output. Distributed jobs may store an exception for each task. The CAC-provided function geterrors(j) collects the errors from the tasks of a job and pretty-prints them for you. 60

61 Getting Errors SimpleError.m has a simple error in the task function. Submit SimpleErrorSubmit.m and examine the error: >> [j,a] = SimpleErrorSubmit(sched); >> geterrors(j); % how do we view just one stacktrace? >> ts = get(j,'tasks'); >> es = get(ts,'errors'); >> es.message >> es.stack(1) 61

62 Manual Retrieval Sometimes a job will hang or fail in such a way that the files don t get downloaded from TUC correctly. In this case, you ll want to retrieve those files manually. >> downloadjob(sched,j); Here s what to do first for a job defined in a prior session: >> cacsched % re-create the sched object >> j = sched.jobs(12) % copy the Jobs(12) object, e.g. >> get(j,'name') % check the job s name 62

63 Manual Retrieval with gridftp For large parallel jobs, you may want to use gridftp to shortcut downloadjob, which downloads all of the files. If you need to spot the error in a large parallel job, for example, you can just use gridftp to grab Task1.out.mat. It very likely contains the exception, because the error is almost always found in all Tasks or the master (Task1). >> ftp = gridftp(); >> ftp.get( Job4/Task1.out.mat ); >> ftp.close(); 63

64 Stdout, Stderr For a parallel job named JobN, the standard output and errors for all the MATLAB processes are stored in two files called JobN.ou and JobN.er respectively. In a distributed job, each TaskM of writes its own output and error into TaskM.ou and TaskM.er in the JobN folder. But capturing errors doesn t help you catch other things: Problems with numerics Running times of different sections of your code Progress of a long-running job 64

65 Printf There are two other ways to get diagnostic information: capturecommandwindowoutput - if true for a task, this property tells MATLAB to return output from fprintf statements and other console output (e.g., from statements lacking a semicolon) to the client. littlejohnlog/qpeek - this pair of functions can be used to create a log for a long-running job and examine the log as the job runs. Usage ideas can be found in caclog.m and caclogsubmit.m, in the examples folder. 65

66 Printf Example Verbose.m and VerboseSubmit.m contain both fprintf and littlejohnlog statements. Notice that the tasks must be set up to return the command window output at task creation. Run the jobs and make sure you can retrieve the output manually as well as using the getoutput(job) function. >> at = get(j, Tasks ); >> out = get(at, CommandWindowOutput ); >> getoutput(j); 66

67 Debug Lab Intro The Traveling Salesman Problem is a classic minimization problem. A salesman has a fixed set of cities that he must visit (each only one time). What order of city visits will minimize the total distance travelled? ga_run.m solves this problem using a genetic algorithm (GA) on the airport dataset that we worked with in the file transfer lab. This is a relatively small dataset with about 150 locations. ga_run2.m is a buggy version that solves the problem for a larger dataset (cities.txt). 67

68 Debug Lab Instructions Examine the output from geterrors and getoutput in order to find and fix the problems with ga_run2.m and help our intrepid salesman out. The functions defined in the two.m files take the same arguments and return the same outputs, so ga_submit.m should not need to be modified, except to change the name of the function in createtask. 68