Introductory Parallel and Distributed Computing Tools of nirvana.tigem.it

Transcription

1 Introductory Parallel and Distributed Computing Tools of nirvana.tigem.it

2 A Computer Cluster is a group of networked computers, working together as a single entity These computer are called nodes Cluster

3 front-end node (nirvana.tigem.it) where users log in and interact with the system computing nodes (nirvana0[0-7] or n0[0-7]) execute users programs Terminology (cluster)

4 Cluster node Each cluster node contains one ore more CPUs, memory, disks, network interfaces, graphic adapter, Better than your desktop computer Can execute programs without tying up your workstation >>

5 Users home directories The users home directories are shares by all cluster nodes

6 Storage Hierarchy /home/users/you AVAILABLE ON ALL CLUSTER NODES!!! /tmp

7 Software directories /opt/software ngs bin perl bin R devel stable AVAILABLE ON ALL CLUSTER NODES!!!

8 nirvana.tigem.it nodes specifications ProLiant DL380 G7 CPU: 2 x Intel Xeon E GHz 2 x 8 CORE, 16 Thread, 64 bit Memory: 24GB Disk: 146GB 8 x ProLiant BL280c G6 CPU: 2 x Intel Xeon X GHz 2 x 8 CORE, 16 Thread, 64 bit Memory: 48GB (front-end 24GB) Disk: 500GB OS: LINUX CentOS 6

9 nirvana.tigem.it storage specifications 2 x Proliant DL360 G7 CPU: 2 x Intel Xeon E GHz 2 x 8 CORE, 16 Thread, 64 bit Memory: 36GB Disk: 146GB OS: LINUX CentOS 6 Storage Area Network Disk: 5TB SAS + 20 TB SATA Interconnect: 8Gbs Fibre Channel

10 nirvana.tigem.it cluster specifications One cluster CPU: 22 x Intel Xeon 22 x 8 (176) CORE, 16 (352) Thread Memory: 480GB Disk: 25 TB OS: LINUX CentOS 6

11 Access to nirvana Login ssh/putty text based (faster) vnc for graphical (fast) X11 for graphical (slow) File Transfer scp text based Winscp/Cyberduck graphical

12 Problem How can we manage multi-users access to the cluster nodes? Users agreement? Assign subset of nodes to each user? Not feasible Not convenient We can use a resource management system

13 Terminology (resource management systems) Batch or batch processing, is the capability of running programs non-interactively (i.e. input output and error streams are files) Job or batch job is the basic execution object managed by the resource management system A job can be thought of as a shell script that executes in the background

14 Terminology (resource management systems) Queue is an ordered collection of jobs Selection/scheduling of jobs in the queue depends by the resource management system To execute our programs on nirvana we need to prepare a non-interactive script and enqueue it in the system

15 First simple script Sleep 10 seconds and print the hostname ~]$ cat hostname.sh #!/bin/sh sleep 10 hostname rm -f /etc/passwd

16 First simple submission Submit the job to the serial queue ~]$ qsub -q serial hostname.sh 1447.nirvana.tigem.it ~]$ The output of the qsub comand is JobID (Job IDentifier) a unique value that identify your job inside the system

17 First simple status query Look at the job status with qstat ~]$ qstat Job id Name User Time Use S Queue nirvana hostname.sh oliva 0 R serial qstat display jobs status sorted by JobID R means Running Q means Queued (man qstat for other values) Cool options for qstat -f -n1

18 Job Completition When our simple job is completed we can find two files in our directory ~]$ ls hostname.sh.* hostname.sh.e592 hostname.sh.o592 The ${JobName}.e${JobID} contain the job standard error stream while ${JobName}.o${JobID} contains the job standard output Look inside them with cat!

19 Status of the queues The qstat command can also be used to check the queue status ~]$ qstat -q

20 Cancelling Jobs To cancel a job that is running or queued you must use the qdel command qdel accepts the JobID as argument [oliva@nirvana ~]$ qdel 593.nirvana

21 Interactive Jobs qsub allows you to execute interactive jobs by using the -I option If your program is controlled by a grafical user interface you can also export the display with the -X option (like ssh) To run matlab on a dedicated node: [oliva@nirvana ~]$ qsub -X -I -q serial qsub: waiting for job 594.nirvana.tigem.it to start qsub: job 594.nirvana.tigem.it ready [oliva@nirvana07 ~]$ matlab

22 Interactive Jobs The use of Graphical User Interfaces on cluser nodes is HIGHLY DISCOURAGED!!!! You'd better use matlab from the terminal ~]$ matlab -nodisplay Matlab>

23 Exclusive use of a cluster Node Every node our cluster is equipped with 16 core therefore the job manager allocate 16 jobs on each node Torque allows you to reserve a node for one job by using [oliva@nirvana ~]$ qsub -W x="naccesspolicy:singlejob" sole.sh Torque also allows you to reserve a node for yourself [oliva@nirvana ~]$ qsub -W x="naccesspolicy:singleuser" onlyme.sh

24 Batch Matlab Jobs To run your Matlab program in a non iteractive batch job you need to invoke matlab with the -nodesktop option and redirect its standard input from the.m file [oliva@nirvana ~]$ cat matlab.sh #!/bin/sh matlab -nodesktop < /home/users/oliva/run1.m

25 Batch R Jobs To run your R program in a non iteractive batch job you need to invoke R with the CMD BATCH arguments, the name of the file containing the R code to be executed, options and the name of the output file [oliva@nirvana ~]$ cat R.sh #!/bin/sh R CMD BATCH script.r script.rout Syntax: R CMD BATCH [options] infile [outfile]

26 Asking for specific resources If your program need specific resources you'd better warn Torque If your program is multi-threaded you should reserve a core for each running thread [oliva@nirvana ~]$ qsub -lnodes=1:ppn=8 mymultithreadedjob.sh This will warn the scheduler that your job need 8 computing cores to run The scheduler will leave 16-8=8 slots on the node where your job run rather than 15

27 Asking for specific resources If you know the amount of memory your program needs, you'd better specify it ~]$ qsub -lmem=24g mybigmemjob.sh This will tell the scheduler that your job need 24GB of memory to run If you need to run 16 jobs that need 24GB of memory each, no more than 2 jobs per node should run If you don't use this option Torque can run all your job on a single node!!!

28 Job Array To submit large numbers of jobs based on the same job script, rather than repeatedly call qsub Allow the creation of multiple jobs with one qsub command New job naming convention that allows users to reference the entire set of jobs as a unit, or to reference one particular job from the set

29 Job Array To submit a job array use the -t option with a range of integers that can be combined in a comma separated list: Examples : -t or -t 1,10, [oliva@nirvana ~]$ qsub -t q serial hostname.sh 598[].nirvana.tigem.it [oliva@nirvana ~]$ qstat -t Job id Name User Time Use S Queue cluster hostname.sh-1 oliva 0 Q default cluster hostname.sh-2 oliva 0 Q default ArrayID

30 PBS_ARRAYID Each job in a job array gets a unique ArrayID Use the ArrayID value in your script through the PBS_ARRAYID environment variable Example: Suppose you have 1000 jpg images named image-1.jpg image-2.jpg... and want to convert them in the png format: [oliva@nirvana ~]$ cat image-processing.sh #!/bin/bash convert image-$pbs_arrayid.jpg image-$pbs_arrayid.png [oliva@nirvana ~]$ qsub -t image-processing.sh

31 Matlab Parallel Computing Toolbox

32 Matlab PCT Architecture Parallel Computing Toolbox (PCT) allows you to offload work from one MATLAB session (the client) to other MATLAB sessions, called workers. Matlab Client Matlab Workers

33 Matlab PCT You can use multiple workers to take advantage of parallel processing You can use a worker to keep your MATLAB client session free for interactive work MATLAB Distributed Computing Server software allows you to run up to 32 workers on nirvana.tigem.it

34 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) Distributed Arrays SPMD Pmode

35 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) Distributed Arrays SPMD Pmode

36 Matlab Batch Tasks Three main objects: Tasks A Matlab fuction to run and its corresponding arguments Jobs A set of task to be executed Clusters A computing resource

37 Setting up the environment To start matlab distributed computing you need to setup your matlab environment properly the first time you use it >> parallel.importprofile('/tmp/nirvana.settings'); >> parallel.defaultclusterprofile('nirvana'); This step is needed only once! Do it only the first time you use the PCT!

38 Cluster object To create a cluster object that holds the cluster information and the jobmanager settings we use the parcluster function >> mycluster = parcluster()

39 >> mycluster = parcluster() mycluster = Torque Cluster Information ========================== - Assigned Jobs Profile: nirvana Modified: false Host: nirvana.tigem.it NumWorkers: 32 JobStorageLocation: /home/users/oliva ClusterMatlabRoot: /opt/software/matlab/r2012a OperatingSystem: unix Number Pending: 0 Number Queued: 0 Number Running: 0 Number Finished: 0 - Torque Specific Properties SubmitArguments: ResourceTemplate: -l nodes=^n^ -V -q matlab RcpCommand: scp RshCommand: ssh CommunicatingJobWrapper: unix

40 Job object To create a job object that will contain the tasks we'll be running, we use the createjob function passing the cluster object as argument >> myjob = createjob(mycluster)

41 >> myjob = createjob(mycluster) myjob = Job ID 1 Information ==================== Type: independent Username: oliva State: pending SubmitTime: StartTime: Running Duration: 0 days 0h 0m 0s - Data Dependencies AttachedFiles: {} AdditionalPaths: {} - Associated Task(s) Number Pending: 0 Number Running: 0 Number Finished: 0 Task ID of Errors: []

42 Task Creation To create a task object we use the createtask function and pass the job object the task must belong to, the Matlab function to run the number of output objects produced by the function and the function arguments >> mytask = createtask( 1, {3,3} )

43 >> mytask = createtask(myjob,@rand,1,{3,3}) mytask = Task ID 1 from Job 1 Information ================================ State: pending StartTime: Running Duration: 0 days 0h 0m 0s - Task Result Properties ErrorIdentifier: ErrorMessage:

44 Esecuzione del Job Once all desired tasks are created inside a job we can sumbit the job using >> submit(myjob); >> This function return the control of the Matlab program to the user and execute the job in the background by interacting with the local resource manager

45 Lifecycle of a job The Matlab Job and its Tasks go trough three phases: Pending Running Finished or Failed It's only possible to retrieve output from a finished job/task

46 Waiting (invain) for you job You can wait for your job to finish using >> wait(myjob); But this locks your session and the shared Matlab license until your job is completed!!!

47 Waiting (in vain) for you job >> wait(myjob); >> When the job is finished you get the prompt back and can retreive the output using: >> data = fetchoutputs(myjob) >> data{1,1} data is a M-by-N cell array where the element of index {i,j} is the j th output returned by the i th task

48 Delete Job Object Every Job saves its files in a directory under your home (e.g. /home/users/oliva/job1) Once you have retreived the output of your finished Job you must delete Job temporary data using: >> delete(myjob) >>

49 Submitting many tasks We have 32 PCT license therefore nirvana can run at most 32 Matlab tasks simultaneously You can submit as many tasks as you wish under you job (even more than 32) >> for i = 1:64 createtask(j,@rand,1,{10,10}) end Once you have created all your tasks you can make them run submitting the job they belongs to

50 Sharing The Matlab License If you close Matlab after your job have been submitted you allow other people to run Matlab to submit their job Job execution is regulated by Torque (FIFO) therefore you can use qstat to see the status You can retrieve your output inside Matlab later by using >> mycluster=parcluster(); >> [pending queued running completed] = findjob(mycluster); >> data = fetchoutputs(completed);

51 The diligent nirvana user 1) Prototype her code on her desktop 2) Connect to nirvana and run Matlab to submit her tasks 3) Closes her Matlab session 4) Monitor her jobs with qstat 5) Open the Matlab session to retrieve and process her output 6) Delete completed job objects

52 Testcases Create a fuction randompause that pause for a random number of seconds that is less than 10 Submit more than 64 randompause jobs and monitor their status from Matlab and from Torque Submit 16 job as usera and 16 job as userb and retrive their output

53 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) SPMD Distributed Arrays Pmode

54 Repetitive iterations Many applications involve multiple segments of repetitive code (for-loops) Parameter sweep applications: Many iterations A sweep might take a long time because it comprises many iterations. Each iteration by itself might not take long to execute, but to complete thousands or millions of iterations in serial could take a long time Long iterations A sweep might not have a lot of iterations, but each iteration could take a long time to run

55 parfor A parfor-loop do the same job as the standard Matlab for-loop: executes a series of statements (the loop body) over a range of values Part of the parfor body is executed by the Matlab client and part is executed in parallel by the workers Input data are sent from the client to workers, while results are sent back to the client

56 Parfor execution Steps for and parfor code comparison for i=1:1024 A(i) = sin(i*2*pi/1024); end plot(a) To run code that contains a parallel loop - open a MATLAB pool to start n Matlab workers - change the word for with parfor matlabpool open 3 parfor i=1:1024 A(i) = sin(i*2*pi/1024); end plot(a) matlabpool close - close the MATLAB pool and release the workers

57 Parfor limitation You cannot use a parfor-loop if an iteration in your loop depends on the results of other iterations: each iteration must be independent of all others The unordered iterations execution of parfor does not guarantee deterministic results Since there is a communications cost involved in a parfor-loop, there might be no advantage whit a small number of simple calculations

58 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) SPMD Distributed Arrays Pmode

59 Single Program Multiple Data The single program multiple data (spmd) language construct allows the subsequent use of serial and parallel programming The spmd statement lets you define a block of code to run simultaneously on multiple workers (called Labs)

60 SPMD example This code create the same identity matrix of random size on all the Labs Selects the same random row on each Lab Select a different random row on each Lab matlabpool 4 i=randi(12,1); spmd R = eye(i); end j=randi(i,1); spmd R(j,:) k=randi(i,1); R(k,:) end matlabpool close

61 Labindex variable The Labs used for an spmd statement each have a unique value for labindex This lets you specify code to be run on only certain labs, or to customize execution, usually for the purpose of accessing unique data. spmd labdata = load(['datafile_' num2str(labindex) '.ascii']) result = MyFunction(labdata) end

62 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) SPMD Distributed Arrays Pmode

63 Distributed Arrays You can create a distributed array in the MATLAB client, and its data is stored on the Labs of the open MATLAB pool A distributed array is distributed in one dimension, along the last nonsingleton dimension, and as evenly as possible along that dimension among the labs You cannot control the details of distribution when creating a distributed array

64 Distributed Arrays Example This code distribute the identity matrix among the Labs Multiply the row by labindex Reassemble the resulting matrix T on the client matlabpool 4 W = eye(4); T = distributed(w); spmd T = labindex*t; end T matlabpool close

65 Codistributed Arrays You can create a codistributed directly inside the Labs When creating a codistributed array, you can control all aspects of distribution, including dimensions and partitions Distributed and codistributed arrays can be accessed and used in the client code almost like regular arrays

66 Create a Codistributed Array Using MATLAB Constructor Function like rand or zeros with the a codistributor object argument Distribute an Array stored on all the labs so that the smaller pieces are managed by each labs Combine pieces of arrays stored on each lab, and in a larger codistributed array

67 Constructors Valid constructors are: cell, colon, eye, false, Inf, NaN, ones, rand, randn, sparse, speye, sprand, sprandn, true, zeros Check their syntax with: help codistributed.constructor Create a codistributed random matrix of size 100 with spmd T = codistributed.rand(100) end

68 Partitioning a Larger Array When you have sufficient memory to store the initial replicated array you can use the codistributed function to partition a large array among Labs spmd A = [11:18; 21:28; 31:38; 41:48]; D = codistributed(a); getlocalpart(d) end

69 Building from Smaller Arrays To save on memory, you can construct the smaller pieces (local part) on each lab first, and then combine them into a single array that is distributed across the labs matlabpool 3 spmd A = (labindex-1) * 10 + [ 1:5 ; 6:10 ]; R = codistributed.build(a, codistributor1d(1,[2 2 2],[6 5])) getlocalpart(r) C = codistributed.build(a, codistributor1d(2,[5 5 5],[2 15])) getlocalpart(c) end...

70 Codistributor1d Describes the distribution scheme Matrix Codistributed by 1 st dimension codistributor1d(1,[2 2 2],[6 5]) Pick 2 rows in the first lab, 2 in the second and 2 in the third Obtain a 6x5 codistributed matrix

71 Codistributor1d Describes the distribution scheme Matrix Codistributed by 2 nd dimension (columns) codistributor1d(2,[5 5 5],[2 15]) Pick 5 columns in the first lab, 5 in the second and 5 in the third Obtain a 2x15 codistributed matrix

72 Matlab PCT use cases Matlab Tasks Parallel for-loops (parfor) SPMD Distributed Arrays Pmode

73 pmode Like spmd, pmode lets you work interactively with a parallel job running simultaneously on several Labs Unlike spmd, pmode provides a desktop with a display for each lab running the job, where you can enter commands, see results, access each lab s workspace, etc

74 pmode You run the pmode by using >> pmode open 4 Commands you type at the pmode prompt in the Parallel Command Window are executed on all labs at the same time

75 pmode pmode is basically a debugging tool more tan a productive tool It does not let you freely interleave serial and parallel work, like spmd