Resource Sharing Problem

Transcription

1 1 Resource Management Resource Sharing Problem Locate resources What computers can I use. Which are the IP addresses? Allocate resources Get a free computer now, reserve one for tomorrow at noon Authenticate and access control (authorization) Grid security lecture Prepare a resource for use Free 2MB of disk space, ask the other users to log out Use the resource Run a job (execute an application) Monitor the resource Is the computer still connected to the Internet? Did my application finish? 2 Resource Management Stack 5. Scheduler Optimum resource allocation 4. Resource brokers Map high-level onto local requests Resource matchmaking and negotiation 3. Resource co-allocators Allocate multiple l resources simultaneously 2. Grid site resource manager Bridge over local managers Common Grid resource management language 1. Local resource managers Heterogeneous site autonomy Every local system administrator does what he wants and likes Scheduler Resource Broker Multiple Grid Site Resource Co-allocator Grid Site Resource Manager Local Resource Manager 3 1

2 4 1. Local Resource Management What is a Resource? An entity that is to be shared E.g., computers, networks, storage, data, software, people Does not have to be a physical entity Defined in terms of interfaces,, not devices ssh defines the access to a computer POSIX defines interface to a computer resources Open/close/read/write functions define access to a distributed file system o e.g., NFS, AFS, DFS Everything (or anything) is a resource 5 Compute Resources Single processor computers Home computers Parallel computers Symmetric multi-processors o Set of processors sharing a common main memory Distributed memory computers o Network of workstations CC-NUMA o Physically distributed, logically shared, cache coherent memory Batch queuing systems for controlled use 6 2

3 7 2. Grid Resource Management Why it is Hard? Site autonomy No control over local administration If one site suddenly decides to suspend its daily production it cannot announce the entire Grid At maximum it can publish a local Web page Heterogeneous substrate Many different platforms E.g., queuing systems: LoadLeveler, LSF, PBS, SGE, Co-allocation Simultaneous access to independent resources Unreliability ZID sites should be available at 22:00, but diligent tutors shut them down 8 GRAM Bridge 9 GRAM LL LSF PBS SGE Grid Resource Allocation Manager Grid applications use GRAM to execute processes on computers GRAM uses the local job queuing system to manage resources Set-up by the local system administrator 3

4 10 Grid Resource Allocation Manager GRAM Defines resource layer protocols and APIs that enable clients to securely instantiate a Grid computational task Remote job submissions Relies on local resource management interfaces Load Leveler, LSF, PBS, SGE GSI enabled This is a new required and important Grid feature Wrapper over local resource management (job queuing) systems GRAM Architecture 11 GRAM Client MDS client API calls to locate resources GRAM client API calls to request resource allocation and process creation. Site boundary Gatekeeper Authentication Globus Security Infrastructure Create Request Job Manager Parse RSL Library MDS Update MDS with resource state information Gram Reporter Query current status t of resource Local Resource Manager Monitor & control Process Process Process Allocate & create processes Host Certificates Each Grid host runs a GRAM gatekeeper Each Grid host has a X.509 certificate Identity + Public key /etc/grid-security/hostcert.pem The certificate must be for a machine which has a consistent name in DNS To not be run on a computer using DHCP where a different name could be assigned to your computer Used for mutual authentication with every user 12 4

5 13 Mutual Authentication Protocol User authenticates to GRAM with a proxy certificate GRAM authenticates to the user with a host certificate Handshaking gprotocol Generate a random message Encrypt with the public key partner (taken from the X.509 certificate) Send the encrypted message to the partner The partner decrypts it and sends it back If the decrypted message matches the original random one, the handshaking is completed Host X.509 User Proxy private key Proxy X.509 GRAM Gatekeeper Host private key GRAM Architecture 14 globus-job-run 15 radu@petzeck.dps.uibk.ac.at:~$ grid-proxy-init Your identity: /O=AustrianGrid/O=UIBK/OU=DPS/CN=Radu Prodan Enter GRID pass phrase for this identity: p p y Creating proxy... Done Your proxy is valid until: Thu Feb 3 21:58: radu@petzeck:~$ globus-job-run gescher.vcpc.univie.ac.at /bin/echo "Hello Grid" Hello Grid 5

6 Limitation Full path to executable must be given Big drawback in a Grid environment How am I supposed to know the full home path on each remote site? I never logged in on most of the Grid machines Information service 16 radu@petzeck:~$ globus-job-run gescher.vcpc.univie.ac.at echo "Hello Grid" GRAM Job failed because the executable does not exist (error code 5) The globus-job-run Script radu@petzeck:~$ globus-job-run altix1.uibk.ac.at /bin/echo "Hello Grid" Hello Grid 17 radu@petzeck:~$ time globus-job-run altix1.uibk.ac.at /bin/echo "Hello Grid" Hello Grid real user sys 0m4.523s 0m0.630s 0m0.280s GRAM and PBS radu@petzeck:~$ time globus-job-run altix1.uibk.ac.at/jobmanager-pbs /bin/hostname altix1 18 real user sys 0m18.425s 0m0.580s 0m0.220s 6

7 GRAM and SGE time globus-job-run hc-ma.uibk.ac.at/jobmanager-sge /bin/hostname 19 real 0m33.651s user 0m0.620s sys 0m0.230s Job Termination 20 How to detect job termination? Push events Local manager automatically notifies GRAM through callbacks Not supported Pull events GRAM queries the local manager While(1) { getstate(); Sleep(sec); } Compromise accuracy versus busy waiting (sec = 0) No way to know if the application crashed Master Process Pull event Slave Process Batch Queuing System Application Push event 21 Grid Resource Management Latency Executing jobs on the Grid is prone to HUGE latencies 5 seconds mutual authentication 15 seconds job queuing system o It cannot do busy waiting o Ask every 5-10 seconds or so Grid applications must have executions an order of magnitude higher than 20 seconds Otherwise you waste most of your time waiting 7

8 Batch Job Submission Send a batch job to the Grid globus-job-submit altix1.uibk.ac.at/jobmanagerpbs /bin/hostname Receive a URL handler that is unique on the Grid / Internet Use the URL to query the status of the job globus-job-status / ACTIVE DONE 22 Batch Job Submission 23 petzeck:/home/radu>globus-job-submit altix1.uibk.ac.at/jobmanager-pbs /home/cb56/cb561004/a.out petzeck:/home/radu>globus-job-status ACTIVE petzeck:/home/radu>globus-job-status DONE Job Output globus-job-get-output job_id [-f number ] out -err [file] Job_id job URL -out Gets standard output -err Gets standard error -f number gets the last number of lines from stdout / stderr 24 8

9 Cancel a Job globus-job-cancel job_url Cancels (kills) a job Changes the state to FAILED globus-job-clean job_url Remove all the the fl files associated with a job previously started using globus-job-submit All the information concerning the job is lost Must be used after globus-job-get-output 25 Job Cancellation and Cleanup radu@pc6163-c703:~$ globus-job-cancel Are you sure you want to cancel the job now (Y/N)? Y Job canceled. NOTE: You still need to clean files associated with the job by running globus-job-clean <jobid> radu@pc6163-c703:~$ globus-job-clean WARNING: Cleaning a job means: - Kill the job if it still running, and - Remove the cached output on the remote resource 26 Are you sure you want to cleanup the job now (Y/N)? Y Cleanup successful. Executable Staging Copy the file you want to execute to the remote machine Of course, the executable has to be compiled for the remote architecture Not for the local l one globus-job-run altix1.uibk.ac.at stage /bin/ls 27 a.out altix1.jku.austriangrid.at 9

10 28 Cross Platform Executable Staging cb561004]$ globus-job-run agrid1.uibk.ac.at -stage /bin/ls /scratch/cb56/cb561004/.globus/.gass_cache /local/md5/cf/b fcf8d51c3cecf 89c864/md5/4b/fde7a8d6acbd9901f1e1211 d2ff114/data: /scratch/cb56/cb561004/.globus/.gass_cac he/local/md5/cf/b fcf8d51c3ce cf89c864/md5/4b/fde7a8d6acbd9901f1e12 11d2ff114/data: cannot execute binary file 29 Resource Specification Language Grid job requests may be too complex to be specified using batch commands RSL = Common language for specifying complex job requests GRAM service translates this common language g into scheduler specific language E.g., RSL into PBS RSL is based on a set of (attribute=value) pairs & (executable= /bin/echo ) (arguments= Hello Grid ) GRAM service understands a well defined set of attributes Hello Grid RSL Example (directory = /home/radu/) (executable = /bin/echo) (arguments = Hello Grid ) (stdout = std.out) (stderr = std.err) 30 Save this in run.rsl 10

11 The globusrun Script time globusrun -r altix1.jku.austriangrid.at/jobmanager-pbs -f run.rsl globus_gram_client_callback_allow successful GRAM Job submission successful GLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING GLOBUS_GRAM_PROTOCOL_JOB_STATE_ACTIVE GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE 31 real user sys 0m20.150s 0m0.220s 0m0.010s State Transition Diagram 32 GRAM manages asynchronous state change callbacks Execution Attributes (executable = string) Program to run A file path (absolute or relative) URL for file staging o Copy the input file from the remote URL to local file (directory = string) Directory in which to run (default is $HOME) (arguments = arg1 arg2 arg3...) List of string arguments to program (environment = (var1 val1) (var2 val2)) List of environment variable name/value pairs 33 11

12 Input / Output Attributes 34 (stdin = string) Stdin for program A file path (absolute or relative) URL for stdin staging o Automatically get the input from the given URL (stdout = string) (stderr = string) Stdout / stderr for program A file path (absolute or relative) URL for stdout / stderr staging Sample RSL 35 & (executable = my_prog) (directory = /home/radu/my_dir) (arguments = std1.in std2.in std3.in) (environment = (JAVA_HOME /usr/java)/ (PATH /usr/java/bin) ) (LD_LIBRARY_PATH /usr/lib ) ) (stdin = std.in) (stdout = std.out) (stderr = std.err) Execution Time Attributes 36 (maxtime=integer) Maximum wall clock or cpu runtime (GRAM s choice) in minutes (maxwalltime=integer) Maximum wall clock runtime in minutes Wallclock time = total execution time What you see when you look at the watch (maxcputime=integer) Maximum CPU runtime in minutes How much the process sits on the CPU If the execution time is exceeded the job is killed 12

13 Parallel Processing Attributes 37 (count=integer) Number of processors to allocate on a parallel machine Default is 1 (hostcount=integer) number of nodes to distribute the count processes On SMP clusters Count / hostcount processes per node (count = 4) (hostcount = 2) 2 distinct nodes 2 threads per node Memory Attributes (maxmemory=integer) Maximum amount of memory for each process in megabytes (minmemory=integer) Minimum amount of memory for each process in megabytes 38 It would be interested what happens when you try this Job Type (jobtype=value) Predefined job types for easy use Single run a single instance of the program let the program start the other count-1 processes Multiple Start <count> instances of the program using the appropriate scheduling mechanism MPI Run the program using mpirun -np <count> Condor (a very advanced job queuing system) Start a <count> Condor processes running in standard universe 39 13

14 Sequential Process 40 & (jobtype = single) (count = 1) (directory = /home/radu/my_dir) (executable = my_prog) (arguments = std.in) (stdout = std.out) (stderr = std.err) (maxwalltime = 120) (maxcputime = 60) OpenMP Applications & (jobtype = single) (count = 4) (directory = /home/radu/my_dir) (executable = my_prog) (arguments = std.in) (stdout = std.out) (stderr = std.err) (environment = (OMP_GET_NUM_THREADS 4) ) (maxwalltime = 120) (maxcputime = 120) 41 Allocate count (4) processors Run a single instance of the program my_prog Let my_prog start the other count-1 (3) processes MPI Applications & (jobtype = mpi) (count = 4) (directory = /home/radu/my_dir) (executable = my_prog) (arguments = std.in) (stdout = std.out) (stderr = std.err) (maxwalltime = 120) (maxcputime = 120) 42 Automatically run all the count (4) my_prog instances using mpirun 14

15 Queuing System Attributes (project=string) Project (account) against which to charge (queue=string) Queue into which to submit job short, express, long, 43 RSL Substitutions 44 RSL supports simple variable substitutions Substitutions are declared using a list of pairs (rslsubstitution=(sub1 val1)(sub2 val2)) A substitution is invoked with $(SUB) Processing order: Within scope, processed left-to-right, Outer scope processed before inner scope Variable definition can reference previously defined variables File Staging & (executable = my_prog) (directory = /home/radu/my_dir) (arguments = std1.in std2.in std3.in) (rsl_substitution = URL ) (file_stage_in = ( $(URL)/my_in_file my_in_file ) ) (file_stage_out = ( $(URL)/my_out_file my_out_file ) ) (file_clean_up = my_in_file) (stdin = std.in) (stdout = std.out) (stderr = std.err) 45 15

16 Default RSL Substitutions GRAM defines a set of RSL substitutions before processing the job request Useful for portability Machine Information GLOBUS_HOST_MANUFACTURER GLOBUS_HOST_CPUTYPE GLOBUS_HOST_OSNAME GLOBUS_HOST_OSVERSION Paths to Globus GLOBUS_LOCATION Miscellaneous HOME LOGNAME GLOBUS_ID 46 Other Attributes 47 dryrun Simulate what would happen in terms of resource allocation (do not run the job) Useful for testing whether my interface to GRAM is correct scratch_dir Specifies the location to create a scratch subdirectory A SCRATCH_DIRECTORY substitution will be filled with the name of the directory which is created stdout_position / stderr_position Specifies where in the file remote standard output / error streaming should be restarted from. Fault Tolerance Attributes (save_state=yes) saves job state information to a persistent file on disk (restart) Start a new job manager to manage an existing job Searches for a state file saved with save_state Recover from a GRAM crash proxy_timeout Specifies how long (in seconds) before the delegated X509 proxy expires the job manager should exit 48 16

17 Two-Phase Commit 49 Some application need to be executed exactly once (no more, no less) e.g. Bank transfers You ask GRAM to execute something, but after the mutual authentication, the network crashes You get no PENDING / ACTIVE event Did GRAM receive the request or not? Two-phase commit protocol Preparation phase Commit phase (two_phase=<int>) <int> = seconds to wait before job times out 2PC Protocol 50 Fault Tolerant Example 51 & (count=8) (hostcount= 2:ppn=4") (jobtype=mpi) (directory=/home/radu/apps/lapw0/src) (executable=../src/lapw0) (arguments=lapw0.def) (stdout=st.out) (stderr=st.err) (maxwalltime = 60) (maxcputime = 60) (save_state=yes) (two_phase = yes) (proxy_timeout = 10) 17

18 Commodity Grid Kits 52 Grid Applications Java CoG Kit Application Grid Portals High BiologyChemistry Energy Physics Earth Science Grid Portals CoG Portal Services Python CoG Kit Grid Middleware Grid Management Services Advanced CoG Services (File Management, Job Management, Security Management) SWIG Grid Primitives Execution & Security GT3 OGSA Globus Toolkit GT2 GT4 Grid prototyping Web Services CoG Abstraction Layers 53 Nano materials Bio- Disaster Informatics Management Applications Portals Development Support CoG Gridfaces Layer CoG Data and Task Management Layer CoG Abstraction Layer CoG GridIDE CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG CoG GT3 Others GT4 GT2 OGSI WS-RF Condor Unicore SSH Avaki classic SETI Java CoG Kit Java API to the Globus services Download from ~/.globus.cog.properties usercert=/home/radu/.globus/usercert.pem userkey=/home/radu/.globus/userkey.pem proxy=/tmp/x509up_u

19 Poll Event-based Job import org.globus.gram.*; import org.globus.gram.internal.*; 55 class GramPoll { public static void main(string[] args) { String rsl = "&(executable=\"/bin/ls\")(stdout=\"std.out\")"; GramJobRun job = new GramJobRun(rsl,"agrid1.uibk.ac.at"); job.run(); while(job.getstatus()!= GRAMConstants.STATUS_DONE && job.getstatus()!= GRAMConstants.STATUS_FAILED) try { Thread.sleep(1000); System.out.println(job.getStatusAsString()); } catch(interruptedexception ex) { } } } Push Event-based Job 56 import org.globus.gram.internal.*; import org.globus.gram.*; class GramPush implements GramJobListener { private static boolean done = false; public GramPush(String rsl) throws Exception { GramJob job = new GramJob(rsl); job.addlistener(this); job.request("altix1.uibk.ac.at"); } public static void main(string[] args) throws Exception { String rsl = "&(executable=\"/bin/ls\")(stdout=\"std.out\")"; new GramPush(rsl); while(! done) { Thread.sleep(1000); } } public void statuschanged(gramjob job) { if(job.getstatus() == GRAMConstants.STATUS_DONE job.getstatus() == GRAMConstants.STATUS_FAILED) done = true; } } Multiple Grid Site Resource Co-allocation 19

20 Grid Resource Management Architecture 58 Resource Specification Language Broker + Scheduler RSL specialization Application Ground RSL Queries & Info Information Service Co-allocator Grid resource managers Local resource managers Simple ground RSL GRAM GRAM GRAM LSF PBS NQE Resource Co-allocation 59 Run an application on two different computers in different administration domains The computers are not under the control of the same local batch queuing system Problem: The job may start on one system immediately And stays in the queue for 1 day in the other In 1 day the first system may be no longer free PBS LSF Dynamically Updated Request Online Co-allocator DUROC Run jobs on multiple sites Decomposes the job into multiple GRAM requests Rendez-vous barrier with allocating agent What happens if one site is available but the other busy? One site waits at the barrier Extra RSL options to control barrier synchronization inter sub-job communication RSL 1 GRAM 1 RSL DUROC RSL n GRAM n 60 globus_duroc_runtime_barrier Job Job Procs. Procs. 20

21 Co-allocation Example 61 + ( & (resourcemanagercontact="c703-pc450.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 0") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 0) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi1.out) (stderr = mpi1.err) ) ( & (resourcemanagercontact="c703-pc421.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 1") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 1) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi2.out) (stderr = mpi2.err) ) Resource Management Attributes (resourcemanagercontact=string) (resourcemanagername=string) resource manager to which to submit a subjob (label=string) identifier for this subjob (subjobstarttype=value) alters the startup barrier mechanism values are strict-barrier, loose-barrier, no-barrier (subjobcommstype=value) values are blocking-join and independent if value is set to independent, the subjob won t be seen from the other subjobs when doing inter-subjob communication 62 MPI on the Grid MPI successful for parallel programming Grid idea came from HPC community MPI applications that require Grid computing exist Incremental approach to running Grid applications Full portability of existing MPI applications on the Grid 63 21

22 MPICH Architecture MPICH Architecture 64 Us er The MPI i nt er f ace ( def i ned by t he MPI st andar ds) The MPI CH l ayer (impl ement s t he MPI i nt er f ace) Abstract Device Interface (ADI) A Particular Platform MPP SMP Cl ust er MPICH-G2 Implemented as globus2 device in MPICH GSI-based authentication and authorization Executable and IO file staging Communication intra-machine: local MPI implementation inter-machine: TCP/IP socket communication (Globus I/O, GridFTP) limitation: requires public IP addresses for all parallel computer nodes DUROC for resource allocation barrier at the end of MPI_Init 65 MPICH-G2 Architecture 66 22

23 2-Site MPI Latency Program 67 + ( & (resourcemanagercontact="c703-pc450.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 0") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 0) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi1.out) (stderr = mpi1.err) ) ( & (resourcemanagercontact="c703-pc421.uibk.ac.at/jobmanager-pbs") (jobtype = mpi) (label ="subjob 1") (environment =(GLOBUS_DUROC_SUBJOB_INDEX 1) ) (count = 1) (executable = /afs/zid1.uibk.ac.at/home/c703/c703246/teaching/mpi/a.out) (stdout = mpi2.out) (stderr = mpi2.err) ) 68 Best Effort Resource Allocation GRAM does best effort resource allocation No request for resource allocation is rejected Consequently, no level of service is guaranteed E.g., IP networks, time-sharing CPU schedulers, job queuing systems Problem Run Grid application on sites A and B A is immediately available and allocated B will be available only in 1 hour Advance reservation when and for how long a resource will be available and exclusively allocated Ensures that the application starts simultaneously on two sites Quality of service (QoS) GARA 69 General-purpose Architecture for Reservation and Allocation Integrate QoS for different types of resources networks, CPUs, batch job schedulers, disks Mechanisms for advanced and immediate reservations for resources create, modify, cancel reservations RSL extensions Integration with DUROC for CPU reservations No Globus toolkit integration 23

24 GARA RSL Example & (reservation-type=compute) (start-time= ) (duration=3600) (percentcpu=100) 70 & (reservation-type=network) (start-time= ) (duration=3600) (endpoint-a= ) (endpoint-b= ) (directionality=unidirectional-ba) (bandwidth=150) GRAM Summary 71 Client MDS client API calls to locate resources MDS client API calls to get resource info MDS: Grid Index Info Server Site boundary GRAM client API calls to request resource allocation and process creation. Globus Security Infrastructure Create Gatekeeper GRAM client API state change callbacks Job Manager Parse RSL Library MDS: Grid Resource Info Server Query current status of resource Local Resource Manager Allocate & Request create processes Monitor & control Process Process Process Resource Management Stack 72 Scheduler Resource Broker To come Multiple Grid Site Resource Co-allocator Grid Site Resource Manager Done Local Resource Manager 24

25 Bibliography GRAM DUROC fp.globus.org/duroc/function_reference.html or reference MPICH-G2 GARA