1 Resource Management

Transcription

1 Resource Management 1

2 2 Resource Sharing Problem Locate resources What computers can I use. Which are the IP addresses? Allocate resources Get a free computer now, reserve one for tomorrow at noon Authenticate and access control (authorization) i Grid security lecture Prepare a resource for use Free 2MB of disk space, ask the other users to log out Use the resource Run a job (execute an application) Monitor the resource Is the computer still connected to the Internet? Did my application finish?

3 3 Resource Management Stack 5. Scheduler Optimum resource allocation 4. Resource brokers Map high-level onto local requests Resource matchmaking and negotiation 3. Resource co-allocators Allocate multiple resources simultaneously 2. Grid site resource manager Bridge over local managers Common Grid resource management language 1. Local resource managers Heterogeneous site autonomy Every local l system administrator i does what he wants and likes Scheduler Resource Broker Multiple Grid Site Resource Co-allocator Grid Site Resource Manager Local Resource Manager

4 4 1. Local Resource Management

5 5 What is a Resource? An entity that is to be shared E.g., computers, networks, storage, data, software, people Does not have to be a physical entity Defined in terms of interfaces, not devices ssh defines the access to a computer POSIX defines interface to a computer resources Open/close/read/write functions define access to a distributed ib t d file system o e.g., NFS, AFS, DFS Everything (or anything) is a resource

6 6 Compute Resources Single processor computers Home computers Parallel computers Symmetric multi-processors o Set of processors sharing a common main memory Distributed memory computers o Network of workstations CC-NUMA o Physically distributed, ib t d logically ll shared, cache coherent memory Batch queuing systems for controlled use

7 7 2. Grid Resource Management

8 8 Why it is Hard? Site autonomy No control over local administration If one site suddenly decides to suspend its daily production it cannot announce the entire Grid At maximum it can publish a local Web page Heterogeneous substrate Many different platforms E.g., queuing systems: LoadLeveler, LSF, PBS, SGE, Co-allocation Simultaneous access to independent resources Unreliability ZID sites should be available at 22:00, but diligent tutors shut them down

9 9 GRAM Bridge GRAM LL LSF PBS SGE Grid Resource Allocation Manager Grid applications use GRAM to execute processes on computers GRAM uses the local job queuing system to manage resources manage resources Set-up by the local system administrator

10 10 Grid Resource Allocation on Manager GRAM Defines resource layer protocols and APIs that enable clients to securely instantiate a Grid computational task Remote job submissions Relies on local resource management interfaces Load Leveler, LSF, PBS, SGE GSI enabled This is a new required and important Grid feature Wrapper over local resource management (job queuing) systems

11 11 GRAM Architecture GRAM Client Gatekeeper Authentication Globus Security Infrastructure MDS client API calls to locate resources GRAM client API calls to request resource allocation and process creation. Create MDS Site boundary Gram Reporter Request Job Manager Parse RSL Library Update MDS with resource state information Query current status of resource Local Resource Manager Monitor & control Process Process Process Allocate & create processes

12 12 Host Certificates Each Grid host runs a GRAM gatekeeper Each Grid host has a X.509 certificate Identity + Public key /etc/grid-security/hostcert.pem The certificate must be for a machine which has a consistent name in DNS To not be run on a computer using DHCP where a different name could be assigned to your computer Used for mutual authentication with every user

13 13 Mutual Authentication Protocol User authenticates to GRAM with a proxy certificate GRAM authenticates to the user with a host certificate Handshaking protocol Generate a random message Encrypt with the public key partner (taken from the X.509 certificate) Send the encrypted message to the partner The partner decrypts it and sends it back If the decrypted message matches the original random one, the handshaking h is completed Host X.509 User Proxy private key Proxy X.509 GRAM Gatekeeper Host private key

14 GRAM Architecture 14

15 15 globus-job-run p.u.. grid-proxy-init Your identity: /O=AustrianGrid/O=UIBK/OU=DPS/CN=Radu Prodan Enter GRID pass phrase for this identity: Creating proxy... Done Your proxy is valid until: Thu Feb 3 21:58: radu@petzeck:~$ globus-job-run gescher.vcpc.univie.ac.at /bin/echo "Hello Grid" Hello Grid

16 16 Limitation Full path to executable must be given Big drawback in a Grid environment How am I supposed to know the full home path on each remote site? I never logged in on most of the Grid machines Information service radu@petzeck:~$ globus-job-run gescher.vcpc.univie.ac.at echo "Hello Grid" GRAM Job failed because the executable does not exist (error code 5)

17 17 The globus-job-run Script globus-job-run u altix1.uibk.ac.at /bin/echo "Hello Grid" Hello Grid radu@petzeck:~$ time globus-job-run altix1.uibk.ac.at ac at /bin/echo "Hello Grid" Hello Grid real 0m4.523s user 0m0.630s sys 0m0.280s

18 18 GRAM and PBS time globus-job-run altix1.uibk.ac.at/jobmanager-pbs /bin/hostname/h altix1 real 0m18.425s user 0m0.580s0 0 sys 0m0.220s

19 19 GRAM and SGE time globus-job-run hc-ma.uibk.ac.at/jobmanager-sge /bin/hostname/h real 0m33.651s user 0m0.620s sys 0m0.230s0 0

20 20 Job Termination How to detect job termination? Push events Local manager automatically ti notifies GRAM through callbacks Not supported Pull events GRAM queries the local manager While(1) { getstate(); Sleep(sec); } Compromise accuracy versus busy waiting (sec = 0) No way to know if the application crashed Master Process Pull event Slave Process Batch Queuing System Application Push event

21 21 Grid Resource Management nt Latency Executing jobs on the Grid is prone to HUGE latencies 5 seconds mutual authentication 15 seconds job queuing system o It cannot do busy waiting o Ask every 5-10 seconds or so Grid applications must have executions an order of magnitude higher than 20 seconds Otherwise you waste most of your time waiting

22 22 Batch Job Submission Send a batch job to the Grid globus-job-submit altix1.uibk.ac.at/jobmanager- pbs /bi/h /bin/hostname Receive a URL handler that is unique on the Grid / Internet Use the URL to query the status of the job globus-job-status / ACTIVE DONE

23 23 Batch Job Submission petzeck:/home/radu>globus-job-submit / u u m altix1.uibk.ac.at/jobmanager-pbs /home/cb56/cb561004/a.outout petzeck:/home/radu>globus-job-status ACTIVE petzeck:/home/radu>globus-job-status status DONE

24 24 Job Output globus-job-get-output g upu Job_id job URL job_id [-f number ] out -err [file] -out Gets standard d output -err Gets standard error -f number gets the last number of lines from stdout / stderr

25 25 Cancel a Job globus-job-cancel job_url Cancels (kills) a job Changes the state to FAILED globus-job-clean job_url Remove all the the files associated with a job previously started using globus-job-submit All the information concerning the job is lost Must be used after globus-job-get-output

26 26 Job Cancellation and Cleanup globus-job-cancel ibk t:40001/22451/ / Are you sure you want to cancel the job now (Y/N)? Y Job canceled. NOTE: You still need to clean files associated with the job by running globus-job-clean <jobid> radu@pc6163-c703:~$ globus-job-clean WARNING: Cleaning a job means: - Kill the job if it still running, and - Remove the cached output on the remote resource Are you sure you want to cleanup the job now (Y/N)? Y Cleanup successful.

27 27 Executable Staging Copy the file you want to execute to the remote machine Of course, the executable has to be compiled for the remote architecture Not for the local one globus-job-run altix1.uibk.ac.at stage /bin/ls aout a.out altix1.jku.austriangrid.at

28 28 Cross Platform Executable Staging cb561004]$ globus-job-run agrid1.uibk.ac.at -stage /bin/ls /scratch/cb56/cb561004/.globus/.gass_cache /local/md5/cf/b fcf8d51c3cecf 89c864/md5/4b/fde7a8d6acbd9901f1e1211 d2ff114/data: /scratch/cb56/cb561004/.globus/.gass_caccac he/local/md5/cf/b fcf8d51c3ce cf89c864/md5/4b/fde7a8d6acbd9901f1e12 864/ 8d6 bd9901f d2ff114/data: cannot execute binary file

29 29 Resource Specification Language Grid job requests may be too complex to be specified using batch commands RSL = Common language for specifying i complex job requests GRAM service translates this common language into scheduler specific language g E.g., RSL into PBS RSL is based on a set of (attribute=value) pairs & (executable= /bin/echo ) (arguments= Hello Grid ) GRAM service understands a well defined set of GRAM service understands a well defined set of attributes

30 30 Hello Grid RSL Example (directory = /home/radu/) (executable = /bin/echo) (arguments = Hello Grid ) (stdout = std.out) (stderr = std.err) Save this in run.rsl

31 31 The globusrun Script time globusrun -r altix1.jku.austriangrid.at/jobmanager-pbs b -f run.rsl globus_gram_client_callback_allow successful GRAM Job submission successful GLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING GLOBUS_GRAM_PROTOCOL_JOB_STATE_ACTIVE GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE real 0m20.150s user 0m0.220s sys 0m0.010s010s

32 32 State Transition Diagram GRAM manages asynchronous state change Mm g y u g callbacks

33 33 Execution Attributes (executable = string) Program to run A file path (absolute or relative) URL for file staging o Copy the input file from the remote URL to local l file (directory = string) Directory in which to run (default is $HOME) (arguments = arg1 arg2 arg3...) List of string arguments to program (environment = (var1 val1) (var2 val2)) List of environment nm nt variable name/value pairs

34 34 Input / Output Attributes (stdin = string) Stdin for program A file path (absolute or relative) URL for stdin staging o Automatically get the input from the given URL (stdout = string) (stderr = string) Stdout / stderr for program A file path (absolute or relative) URL for stdout / stderr staging

35 35 Sample RSL & (executable = my_prog) (directory = /home/radu/my_dir) (arguments = std1.in std2.in std3.in) (environment = (JAVA_HOME /usr/java) (stdin (stdout (stderr (PATH /usr/java/bin) ) (LD_LIBRARY_PATH LIBRARY /usr/lib ) ) = std.in) = std.out) = std.err)

36 36 Execution Time Attributes (maxtime=integer) m Maximum wall clock or cpu runtime (GRAM s choice) in minutes (maxwalltime=integer) Maximum wall clock runtime in minutes Wallclock time = total execution time What you see when you look at the watch (maxcputime=integer) Maximum CPU runtime in minutes How much the process sits on the CPU If the execution time is exceeded the job is killed

37 37 Parallel Processing Attributes (count=integer) Number of processors to allocate on a parallel machine Default is 1 (hostcount=integer) number of nodes to distribute the count processes On SMP clusters Count / hostcount processes per node (count = 4) (hostcount = 2) 2 distinct nodes 2 threads per node

38 38 Memory Attributes (maxmemory=integer) Maximum amount of memory for each process in megabytes (minmemory=integer) integer) Minimum amount of memory for each process in megabytes It would be interested what happens when you try this

39 39 Job Type (jobtype=value) Predefined job types for easy use Single run a single instance of the program let the program start the other count-1 processes Multiple Start <count> instances of the program using the appropriate scheduling mechanism MPI Run the program using mpirun -np <count> Condor (a very advanced job queuing system) Start a <count> Condor processes running in standard universe

40 40 Sequential Process & (jobtype (count = 1) (directory (executable (arguments (stdout (stderr = single) = /home/radu/my_dir) dir) = my_prog) = std.in) = std.out) = std.err) (maxwalltime = 120) (maxcputime = 60)

41 41 OpenMP Applications & (jobtype = single) (count = 4) (directory = /home/radu/my_dir)/ / di (executable = my_prog) (arguments = std.in) (stdout = std.out) (stderr = std.err) (environment = (OMP_GET_NUM_THREADS 4) ) (maxwalltime = 120) (maxcputime = 120) Allocate count (4) processors Run a single instance of the program my_prog Let my_prog start the other count-1 (3) processes

42 42 MPI Applications & (jobtype = mpi) (count = 4) (directory = /home/radu/my_dir) (executable = my_prog) (arguments = std.in) (stdout = std.out) (stderr = std.err) (maxwalltime = 120) (maxcputime = 120) Automatically run all the count (4) my prog Automatically run all the count (4) my_prog instances using mpirun

43 43 Queuing System Attributes (project=string) Project (account) against which to charge (queue=string) Queue into which to submit job short, express, long,

44 44 RSL Substitutions RSL supports simple variable substitutions Substitutions are declared using a list of pairs (rslsubstitution=(sub1 (SUB1 val1)(sub2 val2)) A substitution st tut is invoked with $(SUB) Processing order: Within scope, processed left-to-right, Outer scope processed before inner scope Variable definition can reference previously defined d variables

45 45 File Staging & (executable = my_prog) (directory = /home/radu/my_dir) (arguments = std1.in std2.in std3.in) (rsl _ substitution = URL ) (file_stage_in _ = ($( $(URL)/my_in_file _ my_in_file _ )) ) (file_stage_out = ( $(URL)/my_out_file my_out_file ) ) (file_clean_up = my_in_file) (stdin = std.in) (stdout = std.out) (stderr = std.err)

46 46 Default RSL Substitutions GRAM defines a set of RSL substitutions before processing the job request Useful for portability Machine Information GLOBUS_HOST_MANUFACTURER _ GLOBUS_HOST_CPUTYPE GLOBUS_HOST_OSNAME GLOBUS_HOST_OSVERSION Paths to Globus GLOBUS_LOCATION Miscellaneous HOME LOGNAME GLOBUS_ID

47 47 Other Attributes dryrun Simulate what would happen in terms of resource allocation (do not run the job) Useful for testing whether my interface to GRAM is correct scratch_dir Specifies the location to create a scratch subdirectory A SCRATCH_DIRECTORY substitution will be filled with the name of the directory which is created stdout_position / stderr_position Specifies where in the file remote standard output / error streaming should be restarted from.

48 48 Fault Tolerance Attributes (save _ state=yes) saves job state information to a persistent file on disk (restart) Start a new job manager to manage an existing job Searches for a state file saved with save_state Recover from a GRAM crash proxy_timeout Specifies how long (in seconds) before the delegated X509 proxy expires the job manager should exit

49 49 Two-Phase Commit Some application need to be executed exactly once (no more, no less) e.g. Bank transfers You ask GRAM to execute something, but after the mutual authentication, the network crashes You get no PENDING / ACTIVE event Did GRAM receive the request or not? Two-phase commit protocol Preparation phase Commit phase (two_phase=<int>) <int> = seconds to wait before job times out

50 2PC Protocol 50

51 51 Fault Tolerant Example & (count=8) (hostcount= 2:ppn=4") (jobtype=mpi) (directory=/home/radu/apps/lapw0/src) (executable=../src/lapw0) (arguments=lapw0.def) (stdout=st.out) (stderr=st.err) (maxwalltime = 60) (maxcputime = 60) (save_state=yes) (two_phase = yes) (proxy_timeout t = 10)

52 52 Commodity Grid Kits Application Grid Portals Grid Applications Java CoG Kit High BiologyChemistry Energy Physics Earth Science Grid Portals CoG Portal Services Python CoG Kit dleware Grid Mid Grid Management Services Advanced CoG Services (File Management, Job Management, Security Management) SWIG Grid Primitives Execution & Security Grid prototyping Web Services GT3 OGSA Globus Toolkit GT2 GT4

53 53 CoG Abstraction Layers Nano Bio- Disaster materials Informatics Management Applications Portals Development Support CoG Gridfaces Layer CoG Data and Task Management Layer CoG Abstraction Layer CoG og GridI IDE CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG GT3 Others GT4 GT2 OGSI WS-RF Condor Unicore SSH Avaki classic SETI

54 54 Java CoG Kit Java API to the Globus services Download from org/index php/table ~/.globus.cog.properties gp p usercert=/home/radu/.globus/usercert.pem userkey=/home/radu/.globus/userkey.pem / / / k proxy=/tmp/x509up_u104 p p_

55 55 Poll Event-based Job import org.globus.gram.*; import org.globus.gram.internal.*; l class GramPoll { public static void main(string[] args) { String rsl = "&(executable=\"/bin/ls\")(stdout=\"std.out\")"; GramJobRun job = new GramJobRun(rsl,"agrid1.uibk.ac.at"); ibk job.run(); while(job.getstatus() ()!= GRAMConstants.STATUS_ DONE && job.getstatus()!= GRAMConstants.STATUS_FAILED) try { Thread.sleep(1000); System.out.println(job.getStatusAsString()); } catch(interruptedexception p ex) { } } }

56 56 Push Event-based Job import org.globus.gram.internal.*; import org.globus.gram.*; g g class GramPush implements GramJobListener { private static boolean done = false; public GramPush(String rsl) throws Exception { GramJob job = new GramJob(rsl); job.addlistener(this); job.request("altix1.uibk.ac.at"); } public static void main(string[] args) throws Exception { String rsl = "&(executable=\"/bin/ls\")(stdout=\"std.out\")"; new GramPush(rsl); while(! done) { Thread.sleep(1000); } } } public void statuschanged(gramjob job) { if(job.getstatus() == GRAMConstants.STATUS_DONE job.getstatus() == GRAMConstants.STATUS_FAILED) done = true; }

57 57 3. Multiple Grid Site Resource Co-allocation

58 Grid Resource Management 58 Architecture t Resource Specification Broker + Language Scheduler RSL specialization Application Ground RSL Queries & Info Information Service Co-allocator Grid resource managers Local resource managers Simple ground RSL GRAM GRAM GRAM LSF PBS NQE

59 59 Resource Co-allocation Run an application on two different computers in different administration domains The computers are not under the control of the same local l batch queuing system Problem: The job may start on one system immediately And stays s in the queue for 1 day in the other I 1 d h fi b In 1 day the first system may be no longer free PBS LSF

60 Dynamically Updated Request 60 DUROC Online Co-allocator Run jobs on multiple sites Decomposes the job into multiple GRAM requests Rendez-vous barrier with allocating agent What happens if one site is available but the other busy? RSL 1 GRAM 1 One site waits at the barrier Job Extra RSL options to control barrier synchronization inter sub-job communication Procs. RSL DUROC RSL n GRAM n globus_duroc_runtime_barrier Job Procs.

61 61 Co-allocation Example + ( & (resourcemanagercontact="c703-pc450 c703 pc450.uibk.ac.at/jobmanager at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 0") (environment =(GLOBUS_DUROC_SUBJOB_INDEX SUBJOB 0) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi1.out) (stderr = mpi1.err) ) ( & (resourcemanagercontact="c703-pc421.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 1") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 1) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (td (stdout t = mpi2.out) (stderr = mpi2.err) )

62 62 Resource Management nt Attributes (resourcemanagercontact=string) g) (resourcemanagername=string) resource manager to which to submit a subjob (label=string) identifier for this subjob (subjobstarttype=value) alters the startup t barrier mechanism values are strict-barrier, loose-barrier, no-barrier (subjobcommstype=value) values are blocking-join and independent if value is set to independent, the subjob won t be seen from the other subjobs when doing inter-subjob communication

63 63 MPI on the Grid MPI successful for parallel programming Grid idea came from HPC community MPI applications that require Grid computing exist Incremental approach to running Grid applications Full portability of existing MPI applications F p y f g pp on the Grid

64 64 MPICH Architecture MPICH Architecture Us er The MPI interface ( def i ned by t he MPI st andar ds) The MPI CH l ayer (impl ements the MPI i nt er f ace) Abst r act Devi ce I nt er f ace ( ADI ) A Particular Platform MPP SMP Cl ust er

65 65 MPICH-G2 Implemented m as globus2 device in MPICH GSI-based authentication and authorization Executable and IO file staging Communication intra-machine: local MPI implementation inter-machine: TCP/IP P socket communication (Globus I/O, GridFTP) limitation: requires public IP addresses for all parallel computer nodes DUROC for resource allocation barrier at the end of MPI_InitI

66 MPICH-G2 Architecture 66

67 67 2-Site MPI Latency Program + ( & (resourcemanagercontact="c703-pc450 c703 pc450.uibk.ac.at/jobmanager at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 0") (environment =(GLOBUS_DUROC_SUBJOB_INDEX SUBJOB 0) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi1.out) (stderr = mpi1.err) ) ( & (resourcemanagercontact="c703-pc421.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 1") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 1) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (td (stdout t = mpi2.out) (stderr = mpi2.err) )

68 68 Best Effort Resource Allocation GRAM does best effort resource allocation No request for resource allocation is rejected Consequently, no level of service is guaranteed E.g., IP networks, time-sharing CPU schedulers, job queuing systems Problem Run Grid application on sites A and B A is immediately available and allocated B will be available only in 1 hour Advance reservation when and for how long a resource will be available and exclusively allocated Ensures that the application starts simultaneously on two sites Quality of service (QoS)

69 69 GARA General-purpose Architecture for Reservation and Allocation Integrate t QoS for different types of resources networks, CPUs, batch job schedulers, disks Mechanisms for advanced and immediate reservations for resources create, modify, cancel reservations RSL extensions Integration with DUROC for CPU reservations No Globus toolkit integration i

70 70 GARA RSL Example & (reservation-type=compute) p (start-time= ) (duration=3600) (percentcpu=100) & (reservation-type=network) type network) (start-time= ) (duration=3600) (endpoint-a= ) (endpoint-b= ) (directionality=unidirectional-ba) (bandwidth=150)

71 71 GRAM Summary Client MDS client API calls to locate resources MDS client API calls to get resource info MDS: Grid Index Info Server Site boundary GRAM client API calls to request resource allocation and process creation. Globus Security Infrastructure Gatekeeper Create GRAM client API state t change callbacks Job Manager Parse RSL Library MDS: Grid Resource Info Server Request Query current status of resource Local Resource Manager Monitor & control Process Process Process Allocate & create processes

72 72 Resource Management Stack Scheduler Resource Broker To come Multiple Grid Site Resource Co-allocator Grid Site Resource Manager Done Local Resource Manager

73 73 Bibliography GRAM DUROC MPICH-G2 GARA